重构主控编排与运行时预热链路,统一工作区提示词/专家调度协议并补齐 wiki 记忆注入与写回闭环。

同时收敛启动与运维脚本默认行为(含 wiki worker)、更新 Admin 可观测性与相关测试,降低首轮时延并提高运行稳定性。

Made-with: Cursor
This commit is contained in:
oliver 2026-04-26 08:34:33 +08:00
parent 4a23b715a2
commit dbbe3add6a
14438 changed files with 2693620 additions and 2546 deletions

View file

@ -0,0 +1,294 @@
## OpenClaw Android App
Status: **extremely alpha**. The app is actively being rebuilt from the ground up.
### Rebuild Checklist
- [x] New 4-step onboarding flow
- [x] Connect tab with `Setup Code` + `Manual` modes
- [x] Encrypted persistence for gateway setup/auth state
- [x] Chat UI restyled
- [x] Settings UI restyled and de-duplicated (gateway controls moved to Connect)
- [x] QR code scanning in onboarding
- [x] Performance improvements
- [x] Streaming support in chat UI
- [x] Request camera/location and other permissions in onboarding/settings flow
- [x] Push notifications for gateway/chat status updates
- [x] Security hardening (biometric lock, token handling, safer defaults)
- [x] Voice tab full functionality
- [x] Screen tab full functionality
- [ ] Full end-to-end QA and release hardening
## Open in Android Studio
- Open the folder `apps/android`.
## Build / Run
```bash
cd apps/android
./gradlew :app:assemblePlayDebug
./gradlew :app:installPlayDebug
./gradlew :app:testPlayDebugUnitTest
cd ../..
bun run android:bundle:release
```
Third-party debug flavor:
```bash
cd apps/android
./gradlew :app:assembleThirdPartyDebug
./gradlew :app:installThirdPartyDebug
./gradlew :app:testThirdPartyDebugUnitTest
```
`bun run android:bundle:release` auto-bumps Android `versionName`/`versionCode` in `apps/android/app/build.gradle.kts`, then builds two signed release bundles:
- Play build: `apps/android/build/release-bundles/openclaw-<version>-play-release.aab`
- Third-party build: `apps/android/build/release-bundles/openclaw-<version>-third-party-release.aab`
Flavor-specific direct Gradle tasks:
```bash
cd apps/android
./gradlew :app:bundlePlayRelease
./gradlew :app:bundleThirdPartyRelease
```
## Kotlin Lint + Format
```bash
pnpm android:lint
pnpm android:format
```
Android framework/resource lint (separate pass):
```bash
pnpm android:lint:android
```
Direct Gradle tasks:
```bash
cd apps/android
./gradlew :app:ktlintCheck :benchmark:ktlintCheck
./gradlew :app:ktlintFormat :benchmark:ktlintFormat
./gradlew :app:lintDebug
```
`gradlew` auto-detects the Android SDK at `~/Library/Android/sdk` (macOS default) if `ANDROID_SDK_ROOT` / `ANDROID_HOME` are unset.
## Macrobenchmark (Startup + Frame Timing)
```bash
cd apps/android
./gradlew :benchmark:connectedDebugAndroidTest
```
Reports are written under:
- `apps/android/benchmark/build/reports/androidTests/connected/`
## Perf CLI (low-noise)
Deterministic startup measurement + hotspot extraction with compact CLI output:
```bash
cd apps/android
./scripts/perf-startup-benchmark.sh
./scripts/perf-startup-hotspots.sh
```
Benchmark script behavior:
- Runs only `StartupMacrobenchmark#coldStartup` (10 iterations).
- Prints median/min/max/COV in one line.
- Writes timestamped snapshot JSON to `apps/android/benchmark/results/`.
- Auto-compares with previous local snapshot (or pass explicit baseline: `--baseline <old-benchmarkData.json>`).
Hotspot script behavior:
- Ensures debug app installed, captures startup `simpleperf` data for `.MainActivity`.
- Prints top DSOs, top symbols, and key app-path clues (Compose/MainActivity/WebView).
- Writes raw `perf.data` path for deeper follow-up if needed.
## Run on a Real Android Phone (USB)
1) On phone, enable **Developer options** + **USB debugging**.
2) Connect by USB and accept the debugging trust prompt on phone.
3) Verify ADB can see the device:
```bash
adb devices -l
```
4) Install + launch debug build:
```bash
pnpm android:install
pnpm android:run
```
If `adb devices -l` shows `unauthorized`, re-plug and accept the trust prompt again.
### USB-only gateway testing (no LAN dependency)
Use `adb reverse` so Android `localhost:18789` tunnels to your laptop `localhost:18789`.
Terminal A (gateway):
```bash
pnpm openclaw gateway --port 18789 --verbose
```
Terminal B (USB tunnel):
```bash
adb reverse tcp:18789 tcp:18789
```
Then in app **Connect → Manual**:
- Host: `127.0.0.1`
- Port: `18789`
- TLS: off
## Hot Reload / Fast Iteration
This app is native Kotlin + Jetpack Compose.
- For Compose UI edits: use Android Studio **Live Edit** on a debug build (works on physical devices; project `minSdk=31` already meets API requirement).
- For many non-structural code/resource changes: use Android Studio **Apply Changes**.
- For structural/native/manifest/Gradle changes: do full reinstall (`pnpm android:run`).
- Canvas web content already supports live reload when loaded from Gateway `__openclaw__/canvas/` (see `docs/platforms/android.md`).
## Connect / Pair
1) Start the gateway (on your main machine):
```bash
pnpm openclaw gateway --port 18789 --verbose
```
2) In the Android app:
- Open the **Connect** tab.
- Use **Setup Code** or **Manual** mode to connect.
3) Approve pairing (on the gateway machine):
```bash
openclaw devices list
openclaw devices approve <requestId>
```
More details: `docs/platforms/android.md`.
## Permissions
- Discovery:
- Android 13+ (`API 33+`): `NEARBY_WIFI_DEVICES`
- Android 12 and below: `ACCESS_FINE_LOCATION` (required for NSD scanning)
- Foreground service notification (Android 13+): `POST_NOTIFICATIONS`
- Camera:
- `CAMERA` for `camera.snap` and `camera.clip`
- `RECORD_AUDIO` for `camera.clip` when `includeAudio=true`
## Google Play Restricted Permissions
As of March 19, 2026, these manifest permissions are the main Google Play policy risk for this app:
- `READ_SMS`
- `SEND_SMS`
- `READ_CALL_LOG`
Why these matter:
- Google Play treats SMS and Call Log access as highly restricted. In most cases, Play only allows them for the default SMS app, default Phone app, default Assistant, or a narrow policy exception.
- Review usually involves a `Permissions Declaration Form`, policy justification, and demo video evidence in Play Console.
- If we want a Play-safe build, these should be the first permissions removed behind a dedicated product flavor / variant.
Current OpenClaw Android implication:
- APK / sideload build can keep SMS and Call Log features.
- Google Play build should exclude SMS send/search and Call Log search unless the product is intentionally positioned and approved as a default-handler exception case.
- The repo now ships this split as Android product flavors:
- `play`: removes `READ_SMS`, `SEND_SMS`, and `READ_CALL_LOG`, and hides SMS / Call Log surfaces in onboarding, settings, and advertised node capabilities.
- `thirdParty`: keeps the full permission set and the existing SMS / Call Log functionality.
Policy links:
- [Google Play SMS and Call Log policy](https://support.google.com/googleplay/android-developer/answer/10208820?hl=en)
- [Google Play sensitive permissions policy hub](https://support.google.com/googleplay/android-developer/answer/16558241)
- [Android default handlers guide](https://developer.android.com/guide/topics/permissions/default-handlers)
Other Play-restricted surfaces to watch if added later:
- `ACCESS_BACKGROUND_LOCATION`
- `MANAGE_EXTERNAL_STORAGE`
- `QUERY_ALL_PACKAGES`
- `REQUEST_INSTALL_PACKAGES`
- `AccessibilityService`
Reference links:
- [Background location policy](https://support.google.com/googleplay/android-developer/answer/9799150)
- [AccessibilityService policy](https://support.google.com/googleplay/android-developer/answer/10964491?hl=en-GB)
- [Photo and Video Permissions policy](https://support.google.com/googleplay/android-developer/answer/14594990)
## Integration Capability Test (Preconditioned)
This suite assumes setup is already done manually. It does **not** install/run/pair automatically.
Pre-req checklist:
1) Gateway is running and reachable from the Android app.
2) Android app is connected to that gateway and `openclaw nodes status` shows it as paired + connected.
3) App stays unlocked and in foreground for the whole run.
4) Open the app **Screen** tab and keep it active during the run (canvas/A2UI commands require the canvas WebView attached there).
5) Grant runtime permissions for capabilities you expect to pass (camera/mic/location/notification listener/location, etc.).
6) No interactive system dialogs should be pending before test start.
7) Canvas host is enabled and reachable from the device (do not run gateway with `OPENCLAW_SKIP_CANVAS_HOST=1`; startup logs should include `canvas host mounted at .../__openclaw__/`).
8) Local operator test client pairing is approved. If first run fails with `pairing required`, approve latest pending device pairing request, then rerun:
9) For A2UI checks, keep the app on **Screen** tab; the node now auto-refreshes canvas capability once on first A2UI reachability failure (TTL-safe retry).
```bash
openclaw devices list
openclaw devices approve --latest
```
Run:
```bash
pnpm android:test:integration
```
Optional overrides:
- `OPENCLAW_ANDROID_GATEWAY_URL=ws://...` (default: from your local OpenClaw config)
- `OPENCLAW_ANDROID_GATEWAY_TOKEN=...`
- `OPENCLAW_ANDROID_GATEWAY_PASSWORD=...`
- `OPENCLAW_ANDROID_NODE_ID=...` or `OPENCLAW_ANDROID_NODE_NAME=...`
What it does:
- Reads `node.describe` command list from the selected Android node.
- Invokes advertised non-interactive commands.
- Skips `screen.record` in this suite (Android requires interactive per-invocation screen-capture consent).
- Asserts command contracts (success or expected deterministic error for safe-invalid calls like `sms.send` and `notifications.actions`).
Common failure quick-fixes:
- `pairing required` before tests start:
- approve pending device pairing (`openclaw devices approve --latest`) and rerun.
- `A2UI host not reachable` / `A2UI_HOST_NOT_CONFIGURED`:
- ensure gateway canvas host is running and reachable, keep the app on the **Screen** tab. The app will auto-refresh canvas capability once; if it still fails, reconnect app and rerun.
- `NODE_BACKGROUND_UNAVAILABLE: canvas unavailable`:
- app is not effectively ready for canvas commands; keep app foregrounded and **Screen** tab active.
## Contributions
This Android app is currently being rebuilt.
Maintainer: @obviyus. For issues/questions/contributions, please open an issue or reach out on Discord.

View file

@ -0,0 +1,93 @@
Copyright 2018 The Manrope Project Authors (https://github.com/sharanda/manrope)
This Font Software is licensed under the SIL Open Font License, Version 1.1.
This license is copied below, and is also available with a FAQ at:
http://scripts.sil.org/OFL
-----------------------------------------------------------
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
-----------------------------------------------------------
PREAMBLE
The goals of the Open Font License (OFL) are to stimulate worldwide
development of collaborative font projects, to support the font creation
efforts of academic and linguistic communities, and to provide a free and
open framework in which fonts may be shared and improved in partnership
with others.
The OFL allows the licensed fonts to be used, studied, modified and
redistributed freely as long as they are not sold by themselves. The
fonts, including any derivative works, can be bundled, embedded,
redistributed and/or sold with any software provided that any reserved
names are not used by derivative works. The fonts and derivatives,
however, cannot be released under any other type of license. The
requirement for fonts to remain under this license does not apply
to any document created using the fonts or their derivatives.
DEFINITIONS
"Font Software" refers to the set of files released by the Copyright
Holder(s) under this license and clearly marked as such. This may
include source files, build scripts and documentation.
"Reserved Font Name" refers to any names specified as such after the
copyright statement(s).
"Original Version" refers to the collection of Font Software components as
distributed by the Copyright Holder(s).
"Modified Version" refers to any derivative made by adding to, deleting,
or substituting -- in part or in whole -- any of the components of the
Original Version, by changing formats or by porting the Font Software to a
new environment.
"Author" refers to any designer, engineer, programmer, technical
writer or other person who contributed to the Font Software.
PERMISSION & CONDITIONS
Permission is hereby granted, free of charge, to any person obtaining
a copy of the Font Software, to use, study, copy, merge, embed, modify,
redistribute, and sell modified and unmodified copies of the Font
Software, subject to the following conditions:
1) Neither the Font Software nor any of its individual components,
in Original or Modified Versions, may be sold by itself.
2) Original or Modified Versions of the Font Software may be bundled,
redistributed and/or sold with any software, provided that each copy
contains the above copyright notice and this license. These can be
included either as stand-alone text files, human-readable headers or
in the appropriate machine-readable metadata fields within text or
binary files as long as those fields can be easily viewed by the user.
3) No Modified Version of the Font Software may use the Reserved Font
Name(s) unless explicit written permission is granted by the corresponding
Copyright Holder. This restriction only applies to the primary font name as
presented to the users.
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
Software shall not be used to promote, endorse or advertise any
Modified Version, except to acknowledge the contribution(s) of the
Copyright Holder(s) and the Author(s) or with their explicit written
permission.
5) The Font Software, modified or unmodified, in part or in whole,
must be distributed entirely under this license, and must not be
distributed under any other license. The requirement for fonts to
remain under this license does not apply to any document created
using the Font Software.
TERMINATION
This license becomes null and void if any of the above conditions are
not met.
DISCLAIMER
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
OTHER DEALINGS IN THE FONT SOFTWARE.

View file

@ -0,0 +1,290 @@
import com.android.build.api.variant.impl.VariantOutputImpl
val dnsjavaInetAddressResolverService = "META-INF/services/java.net.spi.InetAddressResolverProvider"
val androidStoreFile = providers.gradleProperty("OPENCLAW_ANDROID_STORE_FILE").orNull?.takeIf { it.isNotBlank() }
val androidStorePassword = providers.gradleProperty("OPENCLAW_ANDROID_STORE_PASSWORD").orNull?.takeIf { it.isNotBlank() }
val androidKeyAlias = providers.gradleProperty("OPENCLAW_ANDROID_KEY_ALIAS").orNull?.takeIf { it.isNotBlank() }
val androidKeyPassword = providers.gradleProperty("OPENCLAW_ANDROID_KEY_PASSWORD").orNull?.takeIf { it.isNotBlank() }
val resolvedAndroidStoreFile =
androidStoreFile?.let { storeFilePath ->
if (storeFilePath.startsWith("~/")) {
"${System.getProperty("user.home")}/${storeFilePath.removePrefix("~/")}"
} else {
storeFilePath
}
}
val hasAndroidReleaseSigning =
listOf(resolvedAndroidStoreFile, androidStorePassword, androidKeyAlias, androidKeyPassword).all { it != null }
val wantsAndroidReleaseBuild =
gradle.startParameter.taskNames.any { taskName ->
taskName.contains("Release", ignoreCase = true) ||
Regex("""(^|:)(bundle|assemble)$""").containsMatchIn(taskName)
}
if (wantsAndroidReleaseBuild && !hasAndroidReleaseSigning) {
error(
"Missing Android release signing properties. Set OPENCLAW_ANDROID_STORE_FILE, " +
"OPENCLAW_ANDROID_STORE_PASSWORD, OPENCLAW_ANDROID_KEY_ALIAS, and " +
"OPENCLAW_ANDROID_KEY_PASSWORD in ~/.gradle/gradle.properties.",
)
}
plugins {
id("com.android.application")
id("org.jlleitschuh.gradle.ktlint")
id("org.jetbrains.kotlin.plugin.compose")
id("org.jetbrains.kotlin.plugin.serialization")
}
android {
namespace = "ai.openclaw.app"
compileSdk = 36
// Release signing is local-only; keep the keystore path and passwords out of the repo.
signingConfigs {
if (hasAndroidReleaseSigning) {
create("release") {
storeFile = project.file(checkNotNull(resolvedAndroidStoreFile))
storePassword = checkNotNull(androidStorePassword)
keyAlias = checkNotNull(androidKeyAlias)
keyPassword = checkNotNull(androidKeyPassword)
}
}
}
sourceSets {
getByName("main") {
assets.directories.add("../../shared/OpenClawKit/Sources/OpenClawKit/Resources")
}
}
defaultConfig {
applicationId = "ai.openclaw.app"
minSdk = 31
targetSdk = 36
versionCode = 2026042000
versionName = "2026.4.20"
ndk {
// Support all major ABIs — native libs are tiny (~47 KB per ABI)
abiFilters += listOf("armeabi-v7a", "arm64-v8a", "x86", "x86_64")
}
}
flavorDimensions += "store"
productFlavors {
create("play") {
dimension = "store"
buildConfigField("boolean", "OPENCLAW_ENABLE_SMS", "false")
buildConfigField("boolean", "OPENCLAW_ENABLE_CALL_LOG", "false")
}
create("thirdParty") {
dimension = "store"
buildConfigField("boolean", "OPENCLAW_ENABLE_SMS", "true")
buildConfigField("boolean", "OPENCLAW_ENABLE_CALL_LOG", "true")
}
}
buildTypes {
release {
if (hasAndroidReleaseSigning) {
signingConfig = signingConfigs.getByName("release")
}
isMinifyEnabled = true
isShrinkResources = true
ndk {
debugSymbolLevel = "SYMBOL_TABLE"
}
proguardFiles(getDefaultProguardFile("proguard-android-optimize.txt"), "proguard-rules.pro")
}
debug {
isMinifyEnabled = false
}
}
buildFeatures {
compose = true
buildConfig = true
}
compileOptions {
sourceCompatibility = JavaVersion.VERSION_17
targetCompatibility = JavaVersion.VERSION_17
}
packaging {
resources {
excludes +=
setOf(
"/META-INF/{AL2.0,LGPL2.1}",
"/META-INF/*.version",
"/META-INF/LICENSE*.txt",
"DebugProbesKt.bin",
"kotlin-tooling-metadata.json",
"org/bouncycastle/pqc/crypto/picnic/lowmcL1.bin.properties",
"org/bouncycastle/pqc/crypto/picnic/lowmcL3.bin.properties",
"org/bouncycastle/pqc/crypto/picnic/lowmcL5.bin.properties",
"org/bouncycastle/x509/CertPathReviewerMessages*.properties",
)
}
}
lint {
disable +=
setOf(
"AndroidGradlePluginVersion",
"GradleDependency",
"IconLauncherShape",
"NewerVersionAvailable",
)
warningsAsErrors = true
}
testOptions {
unitTests.isIncludeAndroidResources = true
}
}
androidComponents {
onVariants { variant ->
variant.outputs
.filterIsInstance<VariantOutputImpl>()
.forEach { output ->
val versionName = output.versionName.orNull ?: "0"
val buildType = variant.buildType
val flavorName = variant.flavorName?.takeIf { it.isNotBlank() }
val outputFileName =
if (flavorName == null) {
"openclaw-$versionName-$buildType.apk"
} else {
"openclaw-$versionName-$flavorName-$buildType.apk"
}
output.outputFileName = outputFileName
}
}
}
kotlin {
compilerOptions {
jvmTarget.set(org.jetbrains.kotlin.gradle.dsl.JvmTarget.JVM_17)
allWarningsAsErrors.set(true)
}
}
ktlint {
android.set(true)
ignoreFailures.set(false)
filter {
exclude("**/build/**")
}
}
dependencies {
val composeBom = platform("androidx.compose:compose-bom:2026.02.00")
implementation(composeBom)
androidTestImplementation(composeBom)
implementation("androidx.core:core-ktx:1.17.0")
implementation("androidx.lifecycle:lifecycle-runtime-ktx:2.10.0")
implementation("androidx.activity:activity-compose:1.12.2")
implementation("androidx.webkit:webkit:1.15.0")
implementation("androidx.compose.ui:ui")
implementation("androidx.compose.ui:ui-tooling-preview")
implementation("androidx.compose.material3:material3")
// material-icons-extended pulled in full icon set (~20 MB DEX). Only ~18 icons used.
// R8 will tree-shake unused icons when minify is enabled on release builds.
implementation("androidx.compose.material:material-icons-extended")
debugImplementation("androidx.compose.ui:ui-tooling")
// Material Components (XML theme + resources)
implementation("com.google.android.material:material:1.13.0")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.10.2")
implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.10.0")
implementation("androidx.security:security-crypto:1.1.0")
implementation("androidx.exifinterface:exifinterface:1.4.2")
implementation("com.squareup.okhttp3:okhttp:5.3.2")
implementation("org.bouncycastle:bcprov-jdk18on:1.83")
implementation("org.commonmark:commonmark:0.27.1")
implementation("org.commonmark:commonmark-ext-autolink:0.27.1")
implementation("org.commonmark:commonmark-ext-gfm-strikethrough:0.27.1")
implementation("org.commonmark:commonmark-ext-gfm-tables:0.27.1")
implementation("org.commonmark:commonmark-ext-task-list-items:0.27.1")
// CameraX (for node.invoke camera.* parity)
implementation("androidx.camera:camera-core:1.5.2")
implementation("androidx.camera:camera-camera2:1.5.2")
implementation("androidx.camera:camera-lifecycle:1.5.2")
implementation("androidx.camera:camera-video:1.5.2")
implementation("com.google.android.gms:play-services-code-scanner:16.1.0")
// Unicast DNS-SD (Wide-Area Bonjour) for tailnet discovery domains.
implementation("dnsjava:dnsjava:3.6.4")
testImplementation("junit:junit:4.13.2")
testImplementation("org.jetbrains.kotlinx:kotlinx-coroutines-test:1.10.2")
testImplementation("io.kotest:kotest-runner-junit5-jvm:6.1.3")
testImplementation("io.kotest:kotest-assertions-core-jvm:6.1.3")
testImplementation("com.squareup.okhttp3:mockwebserver:5.3.2")
testImplementation("org.robolectric:robolectric:4.16.1")
testRuntimeOnly("org.junit.vintage:junit-vintage-engine:6.0.2")
}
tasks.withType<Test>().configureEach {
useJUnitPlatform()
}
androidComponents {
onVariants(selector().withBuildType("release")) { variant ->
val variantName = variant.name
val variantNameCapitalized = variantName.replaceFirstChar(Char::titlecase)
val stripTaskName = "strip${variantNameCapitalized}DnsjavaServiceDescriptor"
val mergeTaskName = "merge${variantNameCapitalized}JavaResource"
val minifyTaskName = "minify${variantNameCapitalized}WithR8"
val mergedJar =
layout.buildDirectory.file(
"intermediates/merged_java_res/$variantName/$mergeTaskName/base.jar",
)
val stripTask =
tasks.register(stripTaskName) {
inputs.file(mergedJar)
outputs.file(mergedJar)
doLast {
val jarFile = mergedJar.get().asFile
if (!jarFile.exists()) {
return@doLast
}
val unpackDir = temporaryDir.resolve("merged-java-res")
delete(unpackDir)
copy {
from(zipTree(jarFile))
into(unpackDir)
exclude(dnsjavaInetAddressResolverService)
}
delete(jarFile)
ant.invokeMethod(
"zip",
mapOf(
"destfile" to jarFile.absolutePath,
"basedir" to unpackDir.absolutePath,
),
)
}
}
tasks.matching { it.name == mergeTaskName }.configureEach {
finalizedBy(stripTask)
}
tasks.matching { it.name == minifyTaskName }.configureEach {
dependsOn(stripTask)
}
}
}

View file

@ -0,0 +1,8 @@
-dontwarn org.bouncycastle.**
-dontwarn okhttp3.**
-dontwarn okio.**
-dontwarn com.sun.jna.**
-dontwarn javax.naming.**
-dontwarn lombok.Generated
-dontwarn org.slf4j.impl.StaticLoggerBinder
-dontwarn sun.net.spi.nameservice.NameServiceDescriptor

View file

@ -0,0 +1,92 @@
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_DATA_SYNC" />
<uses-permission android:name="android.permission.POST_NOTIFICATIONS" />
<uses-permission
android:name="android.permission.NEARBY_WIFI_DEVICES"
android:usesPermissionFlags="neverForLocation" />
<uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" />
<uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION" />
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.SEND_SMS" />
<uses-permission android:name="android.permission.READ_SMS" />
<uses-permission android:name="android.permission.READ_MEDIA_IMAGES" />
<uses-permission android:name="android.permission.READ_MEDIA_VISUAL_USER_SELECTED" />
<uses-permission
android:name="android.permission.READ_EXTERNAL_STORAGE"
android:maxSdkVersion="32" />
<uses-permission android:name="android.permission.READ_CONTACTS" />
<uses-permission android:name="android.permission.WRITE_CONTACTS" />
<uses-permission android:name="android.permission.READ_CALL_LOG" />
<uses-permission android:name="android.permission.READ_CALENDAR" />
<uses-permission android:name="android.permission.WRITE_CALENDAR" />
<uses-permission android:name="android.permission.ACTIVITY_RECOGNITION" />
<uses-feature
android:name="android.hardware.camera"
android:required="false" />
<uses-feature
android:name="android.hardware.telephony"
android:required="false" />
<queries>
<intent>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent>
</queries>
<application
android:name=".NodeApp"
android:allowBackup="true"
android:dataExtractionRules="@xml/data_extraction_rules"
android:fullBackupContent="@xml/backup_rules"
android:icon="@mipmap/ic_launcher"
android:roundIcon="@mipmap/ic_launcher_round"
android:label="@string/app_name"
android:supportsRtl="true"
android:networkSecurityConfig="@xml/network_security_config"
android:theme="@style/Theme.OpenClawNode">
<service
android:name=".NodeForegroundService"
android:exported="false"
android:foregroundServiceType="dataSync" />
<service
android:name=".node.DeviceNotificationListenerService"
android:label="@string/app_name"
android:permission="android.permission.BIND_NOTIFICATION_LISTENER_SERVICE"
android:exported="false">
<intent-filter>
<action android:name="android.service.notification.NotificationListenerService" />
</intent-filter>
</service>
<provider
android:name="androidx.core.content.FileProvider"
android:authorities="${applicationId}.fileprovider"
android:exported="false"
android:grantUriPermissions="true">
<meta-data
android:name="android.support.FILE_PROVIDER_PATHS"
android:resource="@xml/file_paths" />
</provider>
<activity
android:name=".MainActivity"
android:exported="true"
android:windowSoftInputMode="adjustResize"
android:configChanges="orientation|screenSize|screenLayout|smallestScreenSize|uiMode|density|keyboard|keyboardHidden|navigation">
<meta-data
android:name="android.app.shortcuts"
android:resource="@xml/shortcuts" />
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
<intent-filter>
<action android:name="android.intent.action.ASSIST" />
<category android:name="android.intent.category.DEFAULT" />
</intent-filter>
</activity>
</application>
</manifest>

View file

@ -0,0 +1,43 @@
package ai.openclaw.app
import android.content.Intent
const val actionAskOpenClaw = "ai.openclaw.app.action.ASK_OPENCLAW"
const val extraAssistantPrompt = "prompt"
enum class HomeDestination {
Connect,
Chat,
Voice,
Screen,
Settings,
}
data class AssistantLaunchRequest(
val source: String,
val prompt: String?,
val autoSend: Boolean,
)
fun parseAssistantLaunchIntent(intent: Intent?): AssistantLaunchRequest? {
val action = intent?.action ?: return null
return when (action) {
Intent.ACTION_ASSIST ->
AssistantLaunchRequest(
source = "assist",
prompt = null,
autoSend = false,
)
actionAskOpenClaw -> {
val prompt = intent.getStringExtra(extraAssistantPrompt)?.trim()?.ifEmpty { null }
AssistantLaunchRequest(
source = "app_action",
prompt = prompt,
autoSend = prompt != null,
)
}
else -> null
}
}

View file

@ -0,0 +1,14 @@
package ai.openclaw.app
enum class CameraHudKind {
Photo,
Recording,
Success,
Error,
}
data class CameraHudState(
val token: Long,
val kind: CameraHudKind,
val message: String,
)

View file

@ -0,0 +1,26 @@
package ai.openclaw.app
import android.content.Context
import android.os.Build
import android.provider.Settings
object DeviceNames {
fun bestDefaultNodeName(context: Context): String {
val deviceName =
runCatching {
Settings.Global.getString(context.contentResolver, "device_name")
}
.getOrNull()
?.trim()
.orEmpty()
if (deviceName.isNotEmpty()) return deviceName
val model =
listOfNotNull(Build.MANUFACTURER?.takeIf { it.isNotBlank() }, Build.MODEL?.takeIf { it.isNotBlank() })
.joinToString(" ")
.trim()
return model.ifEmpty { "Android Node" }
}
}

View file

@ -0,0 +1,15 @@
package ai.openclaw.app
enum class LocationMode(val rawValue: String) {
Off("off"),
WhileUsing("whileUsing"),
;
companion object {
fun fromRawValue(raw: String?): LocationMode {
val normalized = raw?.trim()?.lowercase()
if (normalized == "always") return WhileUsing
return entries.firstOrNull { it.rawValue.lowercase() == normalized } ?: Off
}
}
}

View file

@ -0,0 +1,85 @@
package ai.openclaw.app
import android.os.Bundle
import android.view.WindowManager
import androidx.activity.ComponentActivity
import androidx.activity.compose.setContent
import androidx.activity.viewModels
import androidx.core.view.WindowCompat
import androidx.compose.material3.Surface
import androidx.compose.ui.Modifier
import androidx.lifecycle.Lifecycle
import androidx.lifecycle.lifecycleScope
import androidx.lifecycle.repeatOnLifecycle
import ai.openclaw.app.ui.RootScreen
import ai.openclaw.app.ui.OpenClawTheme
import kotlinx.coroutines.launch
class MainActivity : ComponentActivity() {
private val viewModel: MainViewModel by viewModels()
private lateinit var permissionRequester: PermissionRequester
private var didAttachRuntimeUi = false
private var didStartNodeService = false
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
handleAssistantIntent(intent)
WindowCompat.setDecorFitsSystemWindows(window, false)
permissionRequester = PermissionRequester(this)
lifecycleScope.launch {
repeatOnLifecycle(Lifecycle.State.STARTED) {
viewModel.preventSleep.collect { enabled ->
if (enabled) {
window.addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON)
} else {
window.clearFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON)
}
}
}
}
lifecycleScope.launch {
repeatOnLifecycle(Lifecycle.State.STARTED) {
viewModel.runtimeInitialized.collect { ready ->
if (!ready || didAttachRuntimeUi) return@collect
viewModel.attachRuntimeUi(owner = this@MainActivity, permissionRequester = permissionRequester)
didAttachRuntimeUi = true
if (!didStartNodeService) {
NodeForegroundService.start(this@MainActivity)
didStartNodeService = true
}
}
}
}
setContent {
OpenClawTheme {
Surface(modifier = Modifier) {
RootScreen(viewModel = viewModel)
}
}
}
}
override fun onStart() {
super.onStart()
viewModel.setForeground(true)
}
override fun onStop() {
viewModel.setForeground(false)
super.onStop()
}
override fun onNewIntent(intent: android.content.Intent) {
super.onNewIntent(intent)
setIntent(intent)
handleAssistantIntent(intent)
}
private fun handleAssistantIntent(intent: android.content.Intent?) {
val request = parseAssistantLaunchIntent(intent) ?: return
viewModel.handleAssistantLaunch(request)
}
}

View file

@ -0,0 +1,385 @@
package ai.openclaw.app
import android.app.Application
import androidx.lifecycle.AndroidViewModel
import androidx.lifecycle.LifecycleOwner
import androidx.lifecycle.viewModelScope
import ai.openclaw.app.chat.ChatMessage
import ai.openclaw.app.chat.ChatPendingToolCall
import ai.openclaw.app.chat.ChatSessionEntry
import ai.openclaw.app.chat.OutgoingAttachment
import ai.openclaw.app.gateway.GatewayEndpoint
import ai.openclaw.app.node.CameraCaptureManager
import ai.openclaw.app.node.CanvasController
import ai.openclaw.app.node.SmsManager
import ai.openclaw.app.voice.VoiceConversationEntry
import kotlinx.coroutines.ExperimentalCoroutinesApi
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.SharingStarted
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.flatMapLatest
import kotlinx.coroutines.flow.flowOf
import kotlinx.coroutines.flow.stateIn
@OptIn(ExperimentalCoroutinesApi::class)
class MainViewModel(app: Application) : AndroidViewModel(app) {
private val nodeApp = app as NodeApp
private val prefs = nodeApp.prefs
private val runtimeRef = MutableStateFlow<NodeRuntime?>(null)
private var foreground = true
private val _requestedHomeDestination = MutableStateFlow<HomeDestination?>(null)
val requestedHomeDestination: StateFlow<HomeDestination?> = _requestedHomeDestination
private val _chatDraft = MutableStateFlow<String?>(null)
val chatDraft: StateFlow<String?> = _chatDraft
private val _pendingAssistantAutoSend = MutableStateFlow<String?>(null)
val pendingAssistantAutoSend: StateFlow<String?> = _pendingAssistantAutoSend
private fun ensureRuntime(): NodeRuntime {
runtimeRef.value?.let { return it }
val runtime = nodeApp.ensureRuntime()
runtime.setForeground(foreground)
runtimeRef.value = runtime
return runtime
}
private fun <T> runtimeState(
initial: T,
selector: (NodeRuntime) -> StateFlow<T>,
): StateFlow<T> =
runtimeRef
.flatMapLatest { runtime -> runtime?.let(selector) ?: flowOf(initial) }
.stateIn(viewModelScope, SharingStarted.Eagerly, initial)
val runtimeInitialized: StateFlow<Boolean> =
runtimeRef
.flatMapLatest { runtime -> flowOf(runtime != null) }
.stateIn(viewModelScope, SharingStarted.Eagerly, false)
val canvasCurrentUrl: StateFlow<String?> = runtimeState(initial = null) { it.canvas.currentUrl }
val canvasA2uiHydrated: StateFlow<Boolean> = runtimeState(initial = false) { it.canvasA2uiHydrated }
val canvasRehydratePending: StateFlow<Boolean> = runtimeState(initial = false) { it.canvasRehydratePending }
val canvasRehydrateErrorText: StateFlow<String?> = runtimeState(initial = null) { it.canvasRehydrateErrorText }
val gateways: StateFlow<List<GatewayEndpoint>> = runtimeState(initial = emptyList()) { it.gateways }
val discoveryStatusText: StateFlow<String> = runtimeState(initial = "Searching…") { it.discoveryStatusText }
val notificationForwardingEnabled: StateFlow<Boolean> = prefs.notificationForwardingEnabled
val notificationForwardingMode: StateFlow<NotificationPackageFilterMode> =
prefs.notificationForwardingMode
val notificationForwardingPackages: StateFlow<Set<String>> = prefs.notificationForwardingPackages
val notificationForwardingQuietHoursEnabled: StateFlow<Boolean> =
prefs.notificationForwardingQuietHoursEnabled
val notificationForwardingQuietStart: StateFlow<String> = prefs.notificationForwardingQuietStart
val notificationForwardingQuietEnd: StateFlow<String> = prefs.notificationForwardingQuietEnd
val notificationForwardingMaxEventsPerMinute: StateFlow<Int> =
prefs.notificationForwardingMaxEventsPerMinute
val notificationForwardingSessionKey: StateFlow<String?> = prefs.notificationForwardingSessionKey
val isConnected: StateFlow<Boolean> = runtimeState(initial = false) { it.isConnected }
val isNodeConnected: StateFlow<Boolean> = runtimeState(initial = false) { it.nodeConnected }
val statusText: StateFlow<String> = runtimeState(initial = "Offline") { it.statusText }
val serverName: StateFlow<String?> = runtimeState(initial = null) { it.serverName }
val remoteAddress: StateFlow<String?> = runtimeState(initial = null) { it.remoteAddress }
val pendingGatewayTrust: StateFlow<NodeRuntime.GatewayTrustPrompt?> = runtimeState(initial = null) { it.pendingGatewayTrust }
val seamColorArgb: StateFlow<Long> = runtimeState(initial = 0xFF0EA5E9) { it.seamColorArgb }
val mainSessionKey: StateFlow<String> = runtimeState(initial = "main") { it.mainSessionKey }
val cameraHud: StateFlow<CameraHudState?> = runtimeState(initial = null) { it.cameraHud }
val cameraFlashToken: StateFlow<Long> = runtimeState(initial = 0L) { it.cameraFlashToken }
val instanceId: StateFlow<String> = prefs.instanceId
val displayName: StateFlow<String> = prefs.displayName
val cameraEnabled: StateFlow<Boolean> = prefs.cameraEnabled
val locationMode: StateFlow<LocationMode> = prefs.locationMode
val locationPreciseEnabled: StateFlow<Boolean> = prefs.locationPreciseEnabled
val preventSleep: StateFlow<Boolean> = prefs.preventSleep
val manualEnabled: StateFlow<Boolean> = prefs.manualEnabled
val manualHost: StateFlow<String> = prefs.manualHost
val manualPort: StateFlow<Int> = prefs.manualPort
val manualTls: StateFlow<Boolean> = prefs.manualTls
val gatewayToken: StateFlow<String> = prefs.gatewayToken
val gatewayBootstrapToken: StateFlow<String> = prefs.gatewayBootstrapToken
val onboardingCompleted: StateFlow<Boolean> = prefs.onboardingCompleted
val canvasDebugStatusEnabled: StateFlow<Boolean> = prefs.canvasDebugStatusEnabled
val speakerEnabled: StateFlow<Boolean> = prefs.speakerEnabled
val micEnabled: StateFlow<Boolean> = prefs.talkEnabled
val micCooldown: StateFlow<Boolean> = runtimeState(initial = false) { it.micCooldown }
val micStatusText: StateFlow<String> = runtimeState(initial = "Mic off") { it.micStatusText }
val micLiveTranscript: StateFlow<String?> = runtimeState(initial = null) { it.micLiveTranscript }
val micIsListening: StateFlow<Boolean> = runtimeState(initial = false) { it.micIsListening }
val micQueuedMessages: StateFlow<List<String>> = runtimeState(initial = emptyList()) { it.micQueuedMessages }
val micConversation: StateFlow<List<VoiceConversationEntry>> = runtimeState(initial = emptyList()) { it.micConversation }
val micInputLevel: StateFlow<Float> = runtimeState(initial = 0f) { it.micInputLevel }
val micIsSending: StateFlow<Boolean> = runtimeState(initial = false) { it.micIsSending }
val chatSessionKey: StateFlow<String> = runtimeState(initial = "main") { it.chatSessionKey }
val chatSessionId: StateFlow<String?> = runtimeState(initial = null) { it.chatSessionId }
val chatMessages: StateFlow<List<ChatMessage>> = runtimeState(initial = emptyList()) { it.chatMessages }
val chatError: StateFlow<String?> = runtimeState(initial = null) { it.chatError }
val chatHealthOk: StateFlow<Boolean> = runtimeState(initial = false) { it.chatHealthOk }
val chatThinkingLevel: StateFlow<String> = runtimeState(initial = "off") { it.chatThinkingLevel }
val chatStreamingAssistantText: StateFlow<String?> = runtimeState(initial = null) { it.chatStreamingAssistantText }
val chatPendingToolCalls: StateFlow<List<ChatPendingToolCall>> = runtimeState(initial = emptyList()) { it.chatPendingToolCalls }
val chatSessions: StateFlow<List<ChatSessionEntry>> = runtimeState(initial = emptyList()) { it.chatSessions }
val pendingRunCount: StateFlow<Int> = runtimeState(initial = 0) { it.pendingRunCount }
init {
if (prefs.onboardingCompleted.value) {
ensureRuntime()
}
}
val canvas: CanvasController
get() = ensureRuntime().canvas
val camera: CameraCaptureManager
get() = ensureRuntime().camera
val sms: SmsManager
get() = ensureRuntime().sms
fun attachRuntimeUi(owner: LifecycleOwner, permissionRequester: PermissionRequester) {
val runtime = runtimeRef.value ?: return
runtime.camera.attachLifecycleOwner(owner)
runtime.camera.attachPermissionRequester(permissionRequester)
runtime.sms.attachPermissionRequester(permissionRequester)
}
fun setForeground(value: Boolean) {
foreground = value
val runtime =
if (value && prefs.onboardingCompleted.value) {
ensureRuntime()
} else {
runtimeRef.value
}
runtime?.setForeground(value)
}
fun setDisplayName(value: String) {
prefs.setDisplayName(value)
}
fun setCameraEnabled(value: Boolean) {
prefs.setCameraEnabled(value)
}
fun setLocationMode(mode: LocationMode) {
prefs.setLocationMode(mode)
}
fun setLocationPreciseEnabled(value: Boolean) {
prefs.setLocationPreciseEnabled(value)
}
fun setPreventSleep(value: Boolean) {
prefs.setPreventSleep(value)
}
fun setManualEnabled(value: Boolean) {
prefs.setManualEnabled(value)
}
fun setManualHost(value: String) {
prefs.setManualHost(value)
}
fun setManualPort(value: Int) {
prefs.setManualPort(value)
}
fun setManualTls(value: Boolean) {
prefs.setManualTls(value)
}
fun setGatewayToken(value: String) {
prefs.setGatewayToken(value)
}
fun setGatewayBootstrapToken(value: String) {
prefs.setGatewayBootstrapToken(value)
}
fun setGatewayPassword(value: String) {
prefs.setGatewayPassword(value)
}
fun resetGatewaySetupAuth() {
ensureRuntime().resetGatewaySetupAuth()
}
fun setOnboardingCompleted(value: Boolean) {
if (value) {
ensureRuntime()
}
prefs.setOnboardingCompleted(value)
}
fun setCanvasDebugStatusEnabled(value: Boolean) {
prefs.setCanvasDebugStatusEnabled(value)
}
fun setNotificationForwardingEnabled(value: Boolean) {
ensureRuntime().setNotificationForwardingEnabled(value)
}
fun setNotificationForwardingMode(mode: NotificationPackageFilterMode) {
ensureRuntime().setNotificationForwardingMode(mode)
}
fun setNotificationForwardingPackagesCsv(csv: String) {
val packages =
csv
.split(',')
.map { it.trim() }
.filter { it.isNotEmpty() }
ensureRuntime().setNotificationForwardingPackages(packages)
}
fun setNotificationForwardingQuietHours(
enabled: Boolean,
start: String,
end: String,
): Boolean {
return ensureRuntime().setNotificationForwardingQuietHours(enabled = enabled, start = start, end = end)
}
fun setNotificationForwardingMaxEventsPerMinute(value: Int) {
ensureRuntime().setNotificationForwardingMaxEventsPerMinute(value)
}
fun setNotificationForwardingSessionKey(value: String?) {
ensureRuntime().setNotificationForwardingSessionKey(value)
}
fun setVoiceScreenActive(active: Boolean) {
ensureRuntime().setVoiceScreenActive(active)
}
fun handleAssistantLaunch(request: AssistantLaunchRequest) {
_requestedHomeDestination.value = HomeDestination.Chat
if (request.autoSend) {
_pendingAssistantAutoSend.value = request.prompt
_chatDraft.value = null
return
}
_pendingAssistantAutoSend.value = null
_chatDraft.value = request.prompt
}
fun clearRequestedHomeDestination() {
_requestedHomeDestination.value = null
}
fun clearChatDraft() {
_chatDraft.value = null
}
fun clearPendingAssistantAutoSend() {
_pendingAssistantAutoSend.value = null
}
fun setMicEnabled(enabled: Boolean) {
ensureRuntime().setMicEnabled(enabled)
}
fun setSpeakerEnabled(enabled: Boolean) {
ensureRuntime().setSpeakerEnabled(enabled)
}
fun refreshGatewayConnection() {
ensureRuntime().refreshGatewayConnection()
}
fun connect(endpoint: GatewayEndpoint) {
ensureRuntime().connect(endpoint)
}
fun connect(
endpoint: GatewayEndpoint,
token: String?,
bootstrapToken: String?,
password: String?,
) {
ensureRuntime().connect(
endpoint,
NodeRuntime.GatewayConnectAuth(
token = token,
bootstrapToken = bootstrapToken,
password = password,
),
)
}
fun connectManual() {
ensureRuntime().connectManual()
}
fun disconnect() {
runtimeRef.value?.disconnect()
}
fun acceptGatewayTrustPrompt() {
runtimeRef.value?.acceptGatewayTrustPrompt()
}
fun declineGatewayTrustPrompt() {
runtimeRef.value?.declineGatewayTrustPrompt()
}
fun handleCanvasA2UIActionFromWebView(payloadJson: String) {
ensureRuntime().handleCanvasA2UIActionFromWebView(payloadJson)
}
fun isTrustedCanvasActionUrl(rawUrl: String?): Boolean {
return ensureRuntime().isTrustedCanvasActionUrl(rawUrl)
}
fun requestCanvasRehydrate(source: String = "screen_tab") {
ensureRuntime().requestCanvasRehydrate(source = source, force = true)
}
fun refreshHomeCanvasOverviewIfConnected() {
ensureRuntime().refreshHomeCanvasOverviewIfConnected()
}
fun loadChat(sessionKey: String) {
ensureRuntime().loadChat(sessionKey)
}
fun refreshChat() {
ensureRuntime().refreshChat()
}
fun refreshChatSessions(limit: Int? = null) {
ensureRuntime().refreshChatSessions(limit = limit)
}
fun setChatThinkingLevel(level: String) {
ensureRuntime().setChatThinkingLevel(level)
}
fun switchChatSession(sessionKey: String) {
ensureRuntime().switchChatSession(sessionKey)
}
fun abortChat() {
ensureRuntime().abortChat()
}
fun sendChat(message: String, thinking: String, attachments: List<OutgoingAttachment>) {
ensureRuntime().sendChat(message = message, thinking = thinking, attachments = attachments)
}
suspend fun sendChatAwaitAcceptance(
message: String,
thinking: String,
attachments: List<OutgoingAttachment>,
): Boolean {
return ensureRuntime().sendChatAwaitAcceptance(
message = message,
thinking = thinking,
attachments = attachments,
)
}
}

View file

@ -0,0 +1,37 @@
package ai.openclaw.app
import android.app.Application
import android.os.StrictMode
class NodeApp : Application() {
val prefs: SecurePrefs by lazy { SecurePrefs(this) }
@Volatile private var runtimeInstance: NodeRuntime? = null
fun ensureRuntime(): NodeRuntime {
runtimeInstance?.let { return it }
return synchronized(this) {
runtimeInstance ?: NodeRuntime(this, prefs).also { runtimeInstance = it }
}
}
fun peekRuntime(): NodeRuntime? = runtimeInstance
override fun onCreate() {
super.onCreate()
if (BuildConfig.DEBUG) {
StrictMode.setThreadPolicy(
StrictMode.ThreadPolicy.Builder()
.detectAll()
.penaltyLog()
.build(),
)
StrictMode.setVmPolicy(
StrictMode.VmPolicy.Builder()
.detectAll()
.penaltyLog()
.build(),
)
}
}
}

View file

@ -0,0 +1,162 @@
package ai.openclaw.app
import android.app.Notification
import android.app.NotificationChannel
import android.app.NotificationManager
import android.app.Service
import android.app.PendingIntent
import android.content.Context
import android.content.Intent
import android.content.pm.ServiceInfo
import androidx.core.app.NotificationCompat
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.Job
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.cancel
import kotlinx.coroutines.flow.combine
import kotlinx.coroutines.launch
class NodeForegroundService : Service() {
private val scope: CoroutineScope = CoroutineScope(SupervisorJob() + Dispatchers.Main)
private var notificationJob: Job? = null
private var didStartForeground = false
override fun onCreate() {
super.onCreate()
ensureChannel()
val initial = buildNotification(title = "OpenClaw Node", text = "Starting…")
startForegroundWithTypes(notification = initial)
val runtime = (application as NodeApp).peekRuntime()
if (runtime == null) {
stopSelf()
return
}
notificationJob =
scope.launch {
combine(
runtime.statusText,
runtime.serverName,
runtime.isConnected,
runtime.micEnabled,
runtime.micIsListening,
) { status, server, connected, micEnabled, micListening ->
Quint(status, server, connected, micEnabled, micListening)
}.collect { (status, server, connected, micEnabled, micListening) ->
val title = if (connected) "OpenClaw Node · Connected" else "OpenClaw Node"
val micSuffix =
if (micEnabled) {
if (micListening) " · Mic: Listening" else " · Mic: Pending"
} else {
""
}
val text = (server?.let { "$status · $it" } ?: status) + micSuffix
startForegroundWithTypes(
notification = buildNotification(title = title, text = text),
)
}
}
}
override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
when (intent?.action) {
ACTION_STOP -> {
(application as NodeApp).peekRuntime()?.disconnect()
stopSelf()
return START_NOT_STICKY
}
}
// Keep running; connection is managed by NodeRuntime (auto-reconnect + manual).
return START_STICKY
}
override fun onDestroy() {
notificationJob?.cancel()
scope.cancel()
super.onDestroy()
}
override fun onBind(intent: Intent?) = null
private fun ensureChannel() {
val mgr = getSystemService(NotificationManager::class.java)
val channel =
NotificationChannel(
CHANNEL_ID,
"Connection",
NotificationManager.IMPORTANCE_LOW,
).apply {
description = "OpenClaw node connection status"
setShowBadge(false)
}
mgr.createNotificationChannel(channel)
}
private fun buildNotification(title: String, text: String): Notification {
val launchIntent = Intent(this, MainActivity::class.java).apply {
flags = Intent.FLAG_ACTIVITY_SINGLE_TOP or Intent.FLAG_ACTIVITY_CLEAR_TOP
}
val launchPending =
PendingIntent.getActivity(
this,
1,
launchIntent,
PendingIntent.FLAG_UPDATE_CURRENT or PendingIntent.FLAG_IMMUTABLE,
)
val stopIntent = Intent(this, NodeForegroundService::class.java).setAction(ACTION_STOP)
val stopPending =
PendingIntent.getService(
this,
2,
stopIntent,
PendingIntent.FLAG_UPDATE_CURRENT or PendingIntent.FLAG_IMMUTABLE,
)
return NotificationCompat.Builder(this, CHANNEL_ID)
.setSmallIcon(R.mipmap.ic_launcher)
.setContentTitle(title)
.setContentText(text)
.setContentIntent(launchPending)
.setOngoing(true)
.setOnlyAlertOnce(true)
.setForegroundServiceBehavior(NotificationCompat.FOREGROUND_SERVICE_IMMEDIATE)
.addAction(0, "Disconnect", stopPending)
.build()
}
private fun updateNotification(notification: Notification) {
val mgr = getSystemService(Context.NOTIFICATION_SERVICE) as NotificationManager
mgr.notify(NOTIFICATION_ID, notification)
}
private fun startForegroundWithTypes(notification: Notification) {
if (didStartForeground) {
updateNotification(notification)
return
}
startForeground(NOTIFICATION_ID, notification, ServiceInfo.FOREGROUND_SERVICE_TYPE_DATA_SYNC)
didStartForeground = true
}
companion object {
private const val CHANNEL_ID = "connection"
private const val NOTIFICATION_ID = 1
private const val ACTION_STOP = "ai.openclaw.app.action.STOP"
fun start(context: Context) {
val intent = Intent(context, NodeForegroundService::class.java)
context.startForegroundService(intent)
}
fun stop(context: Context) {
val intent = Intent(context, NodeForegroundService::class.java).setAction(ACTION_STOP)
context.startService(intent)
}
}
}
private data class Quint<A, B, C, D, E>(val first: A, val second: B, val third: C, val fourth: D, val fifth: E)

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,102 @@
package ai.openclaw.app
import java.time.Instant
import java.time.ZoneId
enum class NotificationPackageFilterMode(val rawValue: String) {
Allowlist("allowlist"),
Blocklist("blocklist"),
;
companion object {
fun fromRawValue(raw: String?): NotificationPackageFilterMode {
return entries.firstOrNull { it.rawValue == raw?.trim()?.lowercase() } ?: Blocklist
}
}
}
internal data class NotificationForwardingPolicy(
val enabled: Boolean,
val mode: NotificationPackageFilterMode,
val packages: Set<String>,
val quietHoursEnabled: Boolean,
val quietStart: String,
val quietEnd: String,
val maxEventsPerMinute: Int,
val sessionKey: String?,
)
internal fun NotificationForwardingPolicy.allowsPackage(packageName: String): Boolean {
val normalized = packageName.trim()
if (normalized.isEmpty()) {
return false
}
return when (mode) {
NotificationPackageFilterMode.Allowlist -> packages.contains(normalized)
NotificationPackageFilterMode.Blocklist -> !packages.contains(normalized)
}
}
internal fun NotificationForwardingPolicy.isWithinQuietHours(
nowEpochMs: Long,
zoneId: ZoneId = ZoneId.systemDefault(),
): Boolean {
if (!quietHoursEnabled) {
return false
}
val startMinutes = parseLocalHourMinute(quietStart) ?: return false
val endMinutes = parseLocalHourMinute(quietEnd) ?: return false
if (startMinutes == endMinutes) {
return true
}
val now =
Instant.ofEpochMilli(nowEpochMs)
.atZone(zoneId)
.toLocalTime()
val nowMinutes = now.hour * 60 + now.minute
return if (startMinutes < endMinutes) {
nowMinutes in startMinutes until endMinutes
} else {
nowMinutes >= startMinutes || nowMinutes < endMinutes
}
}
private val localHourMinuteRegex = Regex("""^([01]\d|2[0-3]):([0-5]\d)$""")
internal fun normalizeLocalHourMinute(raw: String): String? {
val trimmed = raw.trim()
val match = localHourMinuteRegex.matchEntire(trimmed) ?: return null
return "${match.groupValues[1]}:${match.groupValues[2]}"
}
internal fun parseLocalHourMinute(raw: String): Int? {
val normalized = normalizeLocalHourMinute(raw) ?: return null
val parts = normalized.split(':')
val hour = parts[0].toInt()
val minute = parts[1].toInt()
return hour * 60 + minute
}
internal class NotificationBurstLimiter {
private val lock = Any()
private var windowStartMs: Long = -1L
private var eventsInWindow: Int = 0
fun allow(nowEpochMs: Long, maxEventsPerMinute: Int): Boolean {
if (maxEventsPerMinute <= 0) {
return false
}
val currentWindow = nowEpochMs - (nowEpochMs % 60_000L)
synchronized(lock) {
if (currentWindow != windowStartMs) {
windowStartMs = currentWindow
eventsInWindow = 0
}
if (eventsInWindow >= maxEventsPerMinute) {
return false
}
eventsInWindow += 1
return true
}
}
}

View file

@ -0,0 +1,200 @@
package ai.openclaw.app
import android.content.pm.PackageManager
import android.content.Intent
import android.Manifest
import android.net.Uri
import android.os.Handler
import android.os.Looper
import android.provider.Settings
import androidx.appcompat.app.AlertDialog
import androidx.activity.ComponentActivity
import androidx.activity.result.ActivityResultLauncher
import androidx.activity.result.contract.ActivityResultContracts
import androidx.core.content.ContextCompat
import androidx.core.app.ActivityCompat
import androidx.lifecycle.Lifecycle
import androidx.lifecycle.LifecycleEventObserver
import kotlinx.coroutines.CompletableDeferred
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.sync.Mutex
import kotlinx.coroutines.sync.withLock
import kotlinx.coroutines.withContext
import kotlinx.coroutines.suspendCancellableCoroutine
import java.util.concurrent.atomic.AtomicBoolean
import kotlin.coroutines.resume
class PermissionRequester(private val activity: ComponentActivity) {
private val mutex = Mutex()
private var pending: CompletableDeferred<Map<String, Boolean>>? = null
private val mainHandler = Handler(Looper.getMainLooper())
private val launcher: ActivityResultLauncher<Array<String>> =
activity.registerForActivityResult(ActivityResultContracts.RequestMultiplePermissions()) { result ->
val p = pending
pending = null
p?.complete(result)
}
suspend fun requestIfMissing(
permissions: List<String>,
timeoutMs: Long = 20_000,
): Map<String, Boolean> =
mutex.withLock {
val missing =
permissions.filter { perm ->
ContextCompat.checkSelfPermission(activity, perm) != PackageManager.PERMISSION_GRANTED
}
if (missing.isEmpty()) {
return permissions.associateWith { true }
}
val needsRationale =
missing.any { ActivityCompat.shouldShowRequestPermissionRationale(activity, it) }
if (needsRationale) {
val proceed = showRationaleDialog(missing)
if (!proceed) {
return permissions.associateWith { perm ->
ContextCompat.checkSelfPermission(activity, perm) == PackageManager.PERMISSION_GRANTED
}
}
}
val deferred = CompletableDeferred<Map<String, Boolean>>()
pending = deferred
withContext(Dispatchers.Main) {
launcher.launch(missing.toTypedArray())
}
val result =
withContext(Dispatchers.Default) {
kotlinx.coroutines.withTimeout(timeoutMs) { deferred.await() }
}
// Merge: if something was already granted, treat it as granted even if launcher omitted it.
val merged =
permissions.associateWith { perm ->
val nowGranted =
ContextCompat.checkSelfPermission(activity, perm) == PackageManager.PERMISSION_GRANTED
result[perm] == true || nowGranted
}
val denied =
merged.filterValues { !it }.keys.filter {
!ActivityCompat.shouldShowRequestPermissionRationale(activity, it)
}
if (denied.isNotEmpty()) {
showSettingsDialog(denied)
}
return merged
}
private suspend fun showRationaleDialog(permissions: List<String>): Boolean =
withContext(Dispatchers.Main) {
if (activity.isFinishing || activity.isDestroyed) {
return@withContext false
}
suspendCancellableCoroutine { cont ->
val lifecycle = activity.lifecycle
var dialog: AlertDialog? = null
var observer: LifecycleEventObserver? = null
val finished = AtomicBoolean(false)
val removeObserver = {
observer?.let(lifecycle::removeObserver)
observer = null
}
fun finish(result: Boolean?) {
if (!finished.compareAndSet(false, true)) return
removeObserver()
dialog?.dismiss()
if (result != null) {
cont.resume(result)
}
}
val actualObserver =
LifecycleEventObserver { _, event ->
if (event != Lifecycle.Event.ON_DESTROY) return@LifecycleEventObserver
finish(false)
}
observer = actualObserver
lifecycle.addObserver(actualObserver)
cont.invokeOnCancellation {
mainHandler.post {
finish(null)
}
}
dialog =
AlertDialog.Builder(activity)
.setTitle("Permission required")
.setMessage(buildRationaleMessage(permissions))
.setPositiveButton("Continue") { _, _ -> finish(true) }
.setNegativeButton("Not now") { _, _ -> finish(false) }
.setOnCancelListener { finish(false) }
.show()
}
}
private suspend fun showSettingsDialog(permissions: List<String>) =
withContext(Dispatchers.Main) {
if (activity.isFinishing || activity.isDestroyed) return@withContext
val lifecycle = activity.lifecycle
var dialog: AlertDialog? = null
var observer: LifecycleEventObserver? = null
val removeObserver = {
observer?.let(lifecycle::removeObserver)
observer = null
}
val actualObserver =
LifecycleEventObserver { _, event ->
if (event != Lifecycle.Event.ON_DESTROY) return@LifecycleEventObserver
removeObserver()
dialog?.dismiss()
}
observer = actualObserver
lifecycle.addObserver(actualObserver)
dialog =
AlertDialog.Builder(activity)
.setTitle("Enable permission in Settings")
.setMessage(buildSettingsMessage(permissions))
.setPositiveButton("Open Settings") { _, _ ->
if (activity.isFinishing || activity.isDestroyed) return@setPositiveButton
val intent =
Intent(
Settings.ACTION_APPLICATION_DETAILS_SETTINGS,
Uri.fromParts("package", activity.packageName, null),
)
activity.startActivity(intent)
}
.setNegativeButton("Cancel", null)
.setOnDismissListener { removeObserver() }
.show()
}
private fun buildRationaleMessage(permissions: List<String>): String {
val labels = permissions.map { permissionLabel(it) }
return "OpenClaw needs ${labels.joinToString(", ")} permissions to continue."
}
private fun buildSettingsMessage(permissions: List<String>): String {
val labels = permissions.map { permissionLabel(it) }
return "Please enable ${labels.joinToString(", ")} in Android Settings to continue."
}
private fun permissionLabel(permission: String): String =
when (permission) {
Manifest.permission.CAMERA -> "Camera"
Manifest.permission.RECORD_AUDIO -> "Microphone"
Manifest.permission.SEND_SMS -> "Send SMS"
Manifest.permission.READ_SMS -> "Read SMS"
Manifest.permission.READ_CONTACTS -> "Read Contacts"
Manifest.permission.WRITE_CONTACTS -> "Write Contacts"
Manifest.permission.READ_CALENDAR -> "Read Calendar"
Manifest.permission.WRITE_CALENDAR -> "Write Calendar"
Manifest.permission.READ_CALL_LOG -> "Read Call Log"
Manifest.permission.ACTIVITY_RECOGNITION -> "Motion Activity"
Manifest.permission.READ_MEDIA_IMAGES -> "Photos"
Manifest.permission.READ_EXTERNAL_STORAGE -> "Photos"
else -> permission
}
}

View file

@ -0,0 +1,553 @@
@file:Suppress("DEPRECATION")
package ai.openclaw.app
import android.content.Context
import android.content.SharedPreferences
import androidx.core.content.edit
import androidx.security.crypto.EncryptedSharedPreferences
import androidx.security.crypto.MasterKey
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonArray
import kotlinx.serialization.json.JsonNull
import kotlinx.serialization.json.JsonPrimitive
import java.util.UUID
class SecurePrefs(
context: Context,
private val securePrefsOverride: SharedPreferences? = null,
) {
companion object {
val defaultWakeWords: List<String> = listOf("openclaw", "claude")
private const val displayNameKey = "node.displayName"
private const val locationModeKey = "location.enabledMode"
private const val voiceWakeModeKey = "voiceWake.mode"
private const val plainPrefsName = "openclaw.node"
private const val securePrefsName = "openclaw.node.secure"
private const val notificationsForwardingEnabledKey = "notifications.forwarding.enabled"
private const val defaultNotificationForwardingEnabled = false
private const val notificationsForwardingModeKey = "notifications.forwarding.mode"
private const val notificationsForwardingPackagesKey = "notifications.forwarding.packages"
private const val notificationsForwardingQuietHoursEnabledKey =
"notifications.forwarding.quietHoursEnabled"
private const val notificationsForwardingQuietStartKey = "notifications.forwarding.quietStart"
private const val notificationsForwardingQuietEndKey = "notifications.forwarding.quietEnd"
private const val notificationsForwardingMaxEventsPerMinuteKey =
"notifications.forwarding.maxEventsPerMinute"
private const val notificationsForwardingSessionKeyKey = "notifications.forwarding.sessionKey"
}
private val appContext = context.applicationContext
private val json = Json { ignoreUnknownKeys = true }
private val plainPrefs: SharedPreferences =
appContext.getSharedPreferences(plainPrefsName, Context.MODE_PRIVATE)
private val masterKey by lazy {
MasterKey.Builder(appContext)
.setKeyScheme(MasterKey.KeyScheme.AES256_GCM)
.build()
}
private val securePrefs: SharedPreferences by lazy { securePrefsOverride ?: createSecurePrefs(appContext, securePrefsName) }
private val _instanceId = MutableStateFlow(loadOrCreateInstanceId())
val instanceId: StateFlow<String> = _instanceId
private val _displayName =
MutableStateFlow(loadOrMigrateDisplayName(context = context))
val displayName: StateFlow<String> = _displayName
private val _cameraEnabled = MutableStateFlow(plainPrefs.getBoolean("camera.enabled", true))
val cameraEnabled: StateFlow<Boolean> = _cameraEnabled
private val _locationMode = MutableStateFlow(loadLocationMode())
val locationMode: StateFlow<LocationMode> = _locationMode
private val _locationPreciseEnabled =
MutableStateFlow(plainPrefs.getBoolean("location.preciseEnabled", true))
val locationPreciseEnabled: StateFlow<Boolean> = _locationPreciseEnabled
private val _preventSleep = MutableStateFlow(plainPrefs.getBoolean("screen.preventSleep", true))
val preventSleep: StateFlow<Boolean> = _preventSleep
private val _manualEnabled =
MutableStateFlow(plainPrefs.getBoolean("gateway.manual.enabled", false))
val manualEnabled: StateFlow<Boolean> = _manualEnabled
private val _manualHost =
MutableStateFlow(plainPrefs.getString("gateway.manual.host", "") ?: "")
val manualHost: StateFlow<String> = _manualHost
private val _manualPort =
MutableStateFlow(plainPrefs.getInt("gateway.manual.port", 18789))
val manualPort: StateFlow<Int> = _manualPort
private val _manualTls =
MutableStateFlow(plainPrefs.getBoolean("gateway.manual.tls", true))
val manualTls: StateFlow<Boolean> = _manualTls
private val _gatewayToken = MutableStateFlow("")
val gatewayToken: StateFlow<String> = _gatewayToken
private val _gatewayBootstrapToken = MutableStateFlow("")
val gatewayBootstrapToken: StateFlow<String> = _gatewayBootstrapToken
private val _onboardingCompleted =
MutableStateFlow(plainPrefs.getBoolean("onboarding.completed", false))
val onboardingCompleted: StateFlow<Boolean> = _onboardingCompleted
private val _lastDiscoveredStableId =
MutableStateFlow(
plainPrefs.getString("gateway.lastDiscoveredStableID", "") ?: "",
)
val lastDiscoveredStableId: StateFlow<String> = _lastDiscoveredStableId
private val _canvasDebugStatusEnabled =
MutableStateFlow(plainPrefs.getBoolean("canvas.debugStatusEnabled", false))
val canvasDebugStatusEnabled: StateFlow<Boolean> = _canvasDebugStatusEnabled
private val _notificationForwardingEnabled =
MutableStateFlow(plainPrefs.getBoolean(notificationsForwardingEnabledKey, defaultNotificationForwardingEnabled))
val notificationForwardingEnabled: StateFlow<Boolean> = _notificationForwardingEnabled
private val _notificationForwardingMode =
MutableStateFlow(
NotificationPackageFilterMode.fromRawValue(
plainPrefs.getString(notificationsForwardingModeKey, null),
),
)
val notificationForwardingMode: StateFlow<NotificationPackageFilterMode> = _notificationForwardingMode
private val _notificationForwardingPackages = MutableStateFlow(loadNotificationForwardingPackages())
val notificationForwardingPackages: StateFlow<Set<String>> = _notificationForwardingPackages
private val storedQuietStart =
normalizeLocalHourMinute(plainPrefs.getString(notificationsForwardingQuietStartKey, "22:00").orEmpty())
?: "22:00"
private val storedQuietEnd =
normalizeLocalHourMinute(plainPrefs.getString(notificationsForwardingQuietEndKey, "07:00").orEmpty())
?: "07:00"
private val storedQuietHoursEnabled =
plainPrefs.getBoolean(notificationsForwardingQuietHoursEnabledKey, false) &&
normalizeLocalHourMinute(plainPrefs.getString(notificationsForwardingQuietStartKey, "22:00").orEmpty()) != null &&
normalizeLocalHourMinute(plainPrefs.getString(notificationsForwardingQuietEndKey, "07:00").orEmpty()) != null
private val _notificationForwardingQuietHoursEnabled =
MutableStateFlow(storedQuietHoursEnabled)
val notificationForwardingQuietHoursEnabled: StateFlow<Boolean> = _notificationForwardingQuietHoursEnabled
private val _notificationForwardingQuietStart = MutableStateFlow(storedQuietStart)
val notificationForwardingQuietStart: StateFlow<String> = _notificationForwardingQuietStart
private val _notificationForwardingQuietEnd = MutableStateFlow(storedQuietEnd)
val notificationForwardingQuietEnd: StateFlow<String> = _notificationForwardingQuietEnd
private val _notificationForwardingMaxEventsPerMinute =
MutableStateFlow(plainPrefs.getInt(notificationsForwardingMaxEventsPerMinuteKey, 20).coerceAtLeast(1))
val notificationForwardingMaxEventsPerMinute: StateFlow<Int> = _notificationForwardingMaxEventsPerMinute
private val _notificationForwardingSessionKey =
MutableStateFlow(
plainPrefs
.getString(notificationsForwardingSessionKeyKey, "")
?.trim()
?.takeIf { it.isNotEmpty() },
)
val notificationForwardingSessionKey: StateFlow<String?> = _notificationForwardingSessionKey
private val _wakeWords = MutableStateFlow(loadWakeWords())
val wakeWords: StateFlow<List<String>> = _wakeWords
private val _voiceWakeMode = MutableStateFlow(loadVoiceWakeMode())
val voiceWakeMode: StateFlow<VoiceWakeMode> = _voiceWakeMode
private val _talkEnabled = MutableStateFlow(plainPrefs.getBoolean("talk.enabled", false))
val talkEnabled: StateFlow<Boolean> = _talkEnabled
private val _speakerEnabled = MutableStateFlow(plainPrefs.getBoolean("voice.speakerEnabled", true))
val speakerEnabled: StateFlow<Boolean> = _speakerEnabled
fun setLastDiscoveredStableId(value: String) {
val trimmed = value.trim()
plainPrefs.edit { putString("gateway.lastDiscoveredStableID", trimmed) }
_lastDiscoveredStableId.value = trimmed
}
fun setDisplayName(value: String) {
val trimmed = value.trim()
plainPrefs.edit { putString(displayNameKey, trimmed) }
_displayName.value = trimmed
}
fun setCameraEnabled(value: Boolean) {
plainPrefs.edit { putBoolean("camera.enabled", value) }
_cameraEnabled.value = value
}
fun setLocationMode(mode: LocationMode) {
plainPrefs.edit { putString(locationModeKey, mode.rawValue) }
_locationMode.value = mode
}
fun setLocationPreciseEnabled(value: Boolean) {
plainPrefs.edit { putBoolean("location.preciseEnabled", value) }
_locationPreciseEnabled.value = value
}
fun setPreventSleep(value: Boolean) {
plainPrefs.edit { putBoolean("screen.preventSleep", value) }
_preventSleep.value = value
}
fun setManualEnabled(value: Boolean) {
plainPrefs.edit { putBoolean("gateway.manual.enabled", value) }
_manualEnabled.value = value
}
fun setManualHost(value: String) {
val trimmed = value.trim()
plainPrefs.edit { putString("gateway.manual.host", trimmed) }
_manualHost.value = trimmed
}
fun setManualPort(value: Int) {
plainPrefs.edit { putInt("gateway.manual.port", value) }
_manualPort.value = value
}
fun setManualTls(value: Boolean) {
plainPrefs.edit { putBoolean("gateway.manual.tls", value) }
_manualTls.value = value
}
fun setGatewayToken(value: String) {
val trimmed = value.trim()
securePrefs.edit { putString("gateway.manual.token", trimmed) }
_gatewayToken.value = trimmed
}
fun setGatewayPassword(value: String) {
saveGatewayPassword(value)
}
fun setGatewayBootstrapToken(value: String) {
saveGatewayBootstrapToken(value)
}
fun setOnboardingCompleted(value: Boolean) {
plainPrefs.edit { putBoolean("onboarding.completed", value) }
_onboardingCompleted.value = value
}
fun setCanvasDebugStatusEnabled(value: Boolean) {
plainPrefs.edit { putBoolean("canvas.debugStatusEnabled", value) }
_canvasDebugStatusEnabled.value = value
}
internal fun getNotificationForwardingPolicy(appPackageName: String): NotificationForwardingPolicy {
val modeRaw = plainPrefs.getString(notificationsForwardingModeKey, null)
val mode = NotificationPackageFilterMode.fromRawValue(modeRaw)
val configuredPackages = loadNotificationForwardingPackages()
val normalizedAppPackage = appPackageName.trim()
val defaultBlockedPackages =
if (normalizedAppPackage.isNotEmpty()) setOf(normalizedAppPackage) else emptySet()
val packages =
when (mode) {
NotificationPackageFilterMode.Allowlist -> configuredPackages
NotificationPackageFilterMode.Blocklist -> configuredPackages + defaultBlockedPackages
}
val maxEvents = plainPrefs.getInt(notificationsForwardingMaxEventsPerMinuteKey, 20)
val quietStart =
normalizeLocalHourMinute(plainPrefs.getString(notificationsForwardingQuietStartKey, "22:00").orEmpty())
?: "22:00"
val quietEnd =
normalizeLocalHourMinute(plainPrefs.getString(notificationsForwardingQuietEndKey, "07:00").orEmpty())
?: "07:00"
val sessionKey =
plainPrefs
.getString(notificationsForwardingSessionKeyKey, "")
?.trim()
?.takeIf { it.isNotEmpty() }
val quietHoursEnabled =
plainPrefs.getBoolean(notificationsForwardingQuietHoursEnabledKey, false) &&
normalizeLocalHourMinute(plainPrefs.getString(notificationsForwardingQuietStartKey, "22:00").orEmpty()) != null &&
normalizeLocalHourMinute(plainPrefs.getString(notificationsForwardingQuietEndKey, "07:00").orEmpty()) != null
return NotificationForwardingPolicy(
enabled = plainPrefs.getBoolean(notificationsForwardingEnabledKey, defaultNotificationForwardingEnabled),
mode = mode,
packages = packages,
quietHoursEnabled = quietHoursEnabled,
quietStart = quietStart,
quietEnd = quietEnd,
maxEventsPerMinute = maxEvents.coerceAtLeast(1),
sessionKey = sessionKey,
)
}
internal fun setNotificationForwardingEnabled(value: Boolean) {
plainPrefs.edit { putBoolean(notificationsForwardingEnabledKey, value) }
_notificationForwardingEnabled.value = value
}
internal fun setNotificationForwardingMode(mode: NotificationPackageFilterMode) {
plainPrefs.edit { putString(notificationsForwardingModeKey, mode.rawValue) }
_notificationForwardingMode.value = mode
}
internal fun setNotificationForwardingPackages(packages: List<String>) {
val sanitized =
packages
.asSequence()
.map { it.trim() }
.filter { it.isNotEmpty() }
.toSet()
.toList()
.sorted()
val encoded = JsonArray(sanitized.map { JsonPrimitive(it) }).toString()
plainPrefs.edit { putString(notificationsForwardingPackagesKey, encoded) }
_notificationForwardingPackages.value = sanitized.toSet()
}
internal fun setNotificationForwardingQuietHours(
enabled: Boolean,
start: String,
end: String,
): Boolean {
if (!enabled) {
plainPrefs.edit { putBoolean(notificationsForwardingQuietHoursEnabledKey, false) }
_notificationForwardingQuietHoursEnabled.value = false
return true
}
val normalizedStart = normalizeLocalHourMinute(start) ?: return false
val normalizedEnd = normalizeLocalHourMinute(end) ?: return false
plainPrefs.edit {
putBoolean(notificationsForwardingQuietHoursEnabledKey, enabled)
putString(notificationsForwardingQuietStartKey, normalizedStart)
putString(notificationsForwardingQuietEndKey, normalizedEnd)
}
_notificationForwardingQuietHoursEnabled.value = enabled
_notificationForwardingQuietStart.value = normalizedStart
_notificationForwardingQuietEnd.value = normalizedEnd
return true
}
internal fun setNotificationForwardingMaxEventsPerMinute(value: Int) {
val normalized = value.coerceAtLeast(1)
plainPrefs.edit {
putInt(notificationsForwardingMaxEventsPerMinuteKey, normalized)
}
_notificationForwardingMaxEventsPerMinute.value = normalized
}
internal fun setNotificationForwardingSessionKey(value: String?) {
val normalized = value?.trim()?.takeIf { it.isNotEmpty() }
plainPrefs.edit {
putString(notificationsForwardingSessionKeyKey, normalized.orEmpty())
}
_notificationForwardingSessionKey.value = normalized
}
fun loadGatewayToken(): String? {
val manual =
_gatewayToken.value.trim().ifEmpty {
val stored = securePrefs.getString("gateway.manual.token", null)?.trim().orEmpty()
if (stored.isNotEmpty()) _gatewayToken.value = stored
stored
}
if (manual.isNotEmpty()) return manual
val key = "gateway.token.${_instanceId.value}"
val stored = securePrefs.getString(key, null)?.trim()
return stored?.takeIf { it.isNotEmpty() }
}
fun saveGatewayToken(token: String) {
val key = "gateway.token.${_instanceId.value}"
securePrefs.edit { putString(key, token.trim()) }
}
fun loadGatewayBootstrapToken(): String? {
val key = "gateway.bootstrapToken.${_instanceId.value}"
val stored =
_gatewayBootstrapToken.value.trim().ifEmpty {
val persisted = securePrefs.getString(key, null)?.trim().orEmpty()
if (persisted.isNotEmpty()) {
_gatewayBootstrapToken.value = persisted
}
persisted
}
return stored.takeIf { it.isNotEmpty() }
}
fun saveGatewayBootstrapToken(token: String) {
val key = "gateway.bootstrapToken.${_instanceId.value}"
val trimmed = token.trim()
securePrefs.edit { putString(key, trimmed) }
_gatewayBootstrapToken.value = trimmed
}
fun loadGatewayPassword(): String? {
val key = "gateway.password.${_instanceId.value}"
val stored = securePrefs.getString(key, null)?.trim()
return stored?.takeIf { it.isNotEmpty() }
}
fun saveGatewayPassword(password: String) {
val key = "gateway.password.${_instanceId.value}"
securePrefs.edit { putString(key, password.trim()) }
}
fun clearGatewaySetupAuth() {
val instanceId = _instanceId.value
securePrefs.edit {
remove("gateway.manual.token")
remove("gateway.token.$instanceId")
remove("gateway.bootstrapToken.$instanceId")
remove("gateway.password.$instanceId")
}
_gatewayToken.value = ""
_gatewayBootstrapToken.value = ""
}
fun loadGatewayTlsFingerprint(stableId: String): String? {
val key = "gateway.tls.$stableId"
return plainPrefs.getString(key, null)?.trim()?.takeIf { it.isNotEmpty() }
}
fun saveGatewayTlsFingerprint(stableId: String, fingerprint: String) {
val key = "gateway.tls.$stableId"
plainPrefs.edit { putString(key, fingerprint.trim()) }
}
fun getString(key: String): String? {
return securePrefs.getString(key, null)
}
fun putString(key: String, value: String) {
securePrefs.edit { putString(key, value) }
}
fun remove(key: String) {
securePrefs.edit { remove(key) }
}
private fun createSecurePrefs(context: Context, name: String): SharedPreferences {
return EncryptedSharedPreferences.create(
context,
name,
masterKey,
EncryptedSharedPreferences.PrefKeyEncryptionScheme.AES256_SIV,
EncryptedSharedPreferences.PrefValueEncryptionScheme.AES256_GCM,
)
}
private fun loadOrCreateInstanceId(): String {
val existing = plainPrefs.getString("node.instanceId", null)?.trim()
if (!existing.isNullOrBlank()) return existing
val fresh = UUID.randomUUID().toString()
plainPrefs.edit { putString("node.instanceId", fresh) }
return fresh
}
private fun loadOrMigrateDisplayName(context: Context): String {
val existing = plainPrefs.getString(displayNameKey, null)?.trim().orEmpty()
if (existing.isNotEmpty() && existing != "Android Node") return existing
val candidate = DeviceNames.bestDefaultNodeName(context).trim()
val resolved = candidate.ifEmpty { "Android Node" }
plainPrefs.edit { putString(displayNameKey, resolved) }
return resolved
}
fun setWakeWords(words: List<String>) {
val sanitized = WakeWords.sanitize(words, defaultWakeWords)
val encoded =
JsonArray(sanitized.map { JsonPrimitive(it) }).toString()
plainPrefs.edit { putString("voiceWake.triggerWords", encoded) }
_wakeWords.value = sanitized
}
fun setVoiceWakeMode(mode: VoiceWakeMode) {
plainPrefs.edit { putString(voiceWakeModeKey, mode.rawValue) }
_voiceWakeMode.value = mode
}
fun setTalkEnabled(value: Boolean) {
plainPrefs.edit { putBoolean("talk.enabled", value) }
_talkEnabled.value = value
}
fun setSpeakerEnabled(value: Boolean) {
plainPrefs.edit { putBoolean("voice.speakerEnabled", value) }
_speakerEnabled.value = value
}
private fun loadNotificationForwardingPackages(): Set<String> {
val raw = plainPrefs.getString(notificationsForwardingPackagesKey, null)?.trim()
if (raw.isNullOrEmpty()) {
return emptySet()
}
return try {
val element = json.parseToJsonElement(raw)
val array = element as? JsonArray ?: return emptySet()
array
.mapNotNull { item ->
when (item) {
is JsonNull -> null
is JsonPrimitive -> item.content.trim().takeIf { it.isNotEmpty() }
else -> null
}
}
.toSet()
} catch (_: Throwable) {
emptySet()
}
}
private fun loadVoiceWakeMode(): VoiceWakeMode {
val raw = plainPrefs.getString(voiceWakeModeKey, null)
val resolved = VoiceWakeMode.fromRawValue(raw)
// Default ON (foreground) when unset.
if (raw.isNullOrBlank()) {
plainPrefs.edit { putString(voiceWakeModeKey, resolved.rawValue) }
}
return resolved
}
private fun loadLocationMode(): LocationMode {
val raw = plainPrefs.getString(locationModeKey, "off")
val resolved = LocationMode.fromRawValue(raw)
if (raw?.trim()?.lowercase() == "always") {
plainPrefs.edit { putString(locationModeKey, resolved.rawValue) }
}
return resolved
}
private fun loadWakeWords(): List<String> {
val raw = plainPrefs.getString("voiceWake.triggerWords", null)?.trim()
if (raw.isNullOrEmpty()) return defaultWakeWords
return try {
val element = json.parseToJsonElement(raw)
val array = element as? JsonArray ?: return defaultWakeWords
val decoded =
array.mapNotNull { item ->
when (item) {
is JsonNull -> null
is JsonPrimitive -> item.content.trim().takeIf { it.isNotEmpty() }
else -> null
}
}
WakeWords.sanitize(decoded, defaultWakeWords)
} catch (_: Throwable) {
defaultWakeWords
}
}
}

View file

@ -0,0 +1,24 @@
package ai.openclaw.app
internal fun normalizeMainKey(raw: String?): String {
val trimmed = raw?.trim()
return if (!trimmed.isNullOrEmpty()) trimmed else "main"
}
internal fun isCanonicalMainSessionKey(raw: String?): Boolean {
val trimmed = raw?.trim().orEmpty()
if (trimmed.isEmpty()) return false
if (trimmed == "global") return true
return trimmed.startsWith("agent:")
}
internal fun resolveAgentIdFromMainSessionKey(raw: String?): String? {
val trimmed = raw?.trim().orEmpty()
if (!trimmed.startsWith("agent:")) return null
return trimmed.removePrefix("agent:").substringBefore(':').trim().ifEmpty { null }
}
internal fun buildNodeMainSessionKey(deviceId: String, agentId: String?): String {
val resolvedAgentId = agentId?.trim().orEmpty().ifEmpty { "main" }
return "agent:$resolvedAgentId:node-${deviceId.take(12)}"
}

View file

@ -0,0 +1,14 @@
package ai.openclaw.app
enum class VoiceWakeMode(val rawValue: String) {
Off("off"),
Foreground("foreground"),
Always("always"),
;
companion object {
fun fromRawValue(raw: String?): VoiceWakeMode {
return entries.firstOrNull { it.rawValue == raw?.trim()?.lowercase() } ?: Foreground
}
}
}

View file

@ -0,0 +1,21 @@
package ai.openclaw.app
object WakeWords {
const val maxWords: Int = 32
const val maxWordLength: Int = 64
fun parseCommaSeparated(input: String): List<String> {
return input.split(",").map { it.trim() }.filter { it.isNotEmpty() }
}
fun parseIfChanged(input: String, current: List<String>): List<String>? {
val parsed = parseCommaSeparated(input)
return if (parsed == current) null else parsed
}
fun sanitize(words: List<String>, defaults: List<String>): List<String> {
val cleaned =
words.map { it.trim() }.filter { it.isNotEmpty() }.take(maxWords).map { it.take(maxWordLength) }
return cleaned.ifEmpty { defaults }
}
}

View file

@ -0,0 +1,641 @@
package ai.openclaw.app.chat
import ai.openclaw.app.gateway.GatewaySession
import java.util.UUID
import java.util.concurrent.ConcurrentHashMap
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Job
import kotlinx.coroutines.delay
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asStateFlow
import kotlinx.coroutines.launch
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonArray
import kotlinx.serialization.json.JsonElement
import kotlinx.serialization.json.JsonNull
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.buildJsonObject
class ChatController(
private val scope: CoroutineScope,
private val session: GatewaySession,
private val json: Json,
private val supportsChatSubscribe: Boolean,
) {
private var appliedMainSessionKey = "main"
private val _sessionKey = MutableStateFlow("main")
val sessionKey: StateFlow<String> = _sessionKey.asStateFlow()
private val _sessionId = MutableStateFlow<String?>(null)
val sessionId: StateFlow<String?> = _sessionId.asStateFlow()
private val _messages = MutableStateFlow<List<ChatMessage>>(emptyList())
val messages: StateFlow<List<ChatMessage>> = _messages.asStateFlow()
private val _errorText = MutableStateFlow<String?>(null)
val errorText: StateFlow<String?> = _errorText.asStateFlow()
private val _healthOk = MutableStateFlow(false)
val healthOk: StateFlow<Boolean> = _healthOk.asStateFlow()
private val _thinkingLevel = MutableStateFlow("off")
val thinkingLevel: StateFlow<String> = _thinkingLevel.asStateFlow()
private val _pendingRunCount = MutableStateFlow(0)
val pendingRunCount: StateFlow<Int> = _pendingRunCount.asStateFlow()
private val _streamingAssistantText = MutableStateFlow<String?>(null)
val streamingAssistantText: StateFlow<String?> = _streamingAssistantText.asStateFlow()
private val pendingToolCallsById = ConcurrentHashMap<String, ChatPendingToolCall>()
private val _pendingToolCalls = MutableStateFlow<List<ChatPendingToolCall>>(emptyList())
val pendingToolCalls: StateFlow<List<ChatPendingToolCall>> = _pendingToolCalls.asStateFlow()
private val _sessions = MutableStateFlow<List<ChatSessionEntry>>(emptyList())
val sessions: StateFlow<List<ChatSessionEntry>> = _sessions.asStateFlow()
private val pendingRuns = mutableSetOf<String>()
private val pendingRunTimeoutJobs = ConcurrentHashMap<String, Job>()
private val pendingRunTimeoutMs = 120_000L
private var lastHealthPollAtMs: Long? = null
fun onDisconnected(message: String) {
_healthOk.value = false
// Not an error; keep connection status in the UI pill.
_errorText.value = null
clearPendingRuns()
pendingToolCallsById.clear()
publishPendingToolCalls()
_streamingAssistantText.value = null
_sessionId.value = null
}
fun load(sessionKey: String) {
val key = normalizeRequestedSessionKey(sessionKey)
_sessionKey.value = key
scope.launch { bootstrap(forceHealth = true, refreshSessions = true) }
}
fun applyMainSessionKey(mainSessionKey: String) {
val trimmed = mainSessionKey.trim()
if (trimmed.isEmpty()) return
val nextState =
applyMainSessionKey(
currentSessionKey = normalizeRequestedSessionKey(_sessionKey.value),
appliedMainSessionKey = appliedMainSessionKey,
nextMainSessionKey = trimmed,
)
appliedMainSessionKey = nextState.appliedMainSessionKey
if (_sessionKey.value == nextState.currentSessionKey) return
_sessionKey.value = nextState.currentSessionKey
scope.launch { bootstrap(forceHealth = true, refreshSessions = true) }
}
fun refresh() {
scope.launch { bootstrap(forceHealth = true, refreshSessions = true) }
}
fun refreshSessions(limit: Int? = null) {
scope.launch { fetchSessions(limit = limit) }
}
fun setThinkingLevel(thinkingLevel: String) {
val normalized = normalizeThinking(thinkingLevel)
if (normalized == _thinkingLevel.value) return
_thinkingLevel.value = normalized
}
fun switchSession(sessionKey: String) {
val key = normalizeRequestedSessionKey(sessionKey)
if (key.isEmpty()) return
if (key == _sessionKey.value) return
_sessionKey.value = key
// Keep the thread switch path lean: history + health are needed immediately,
// but the session list is usually unchanged and can refresh on explicit pull-to-refresh.
scope.launch { bootstrap(forceHealth = true, refreshSessions = false) }
}
private fun normalizeRequestedSessionKey(sessionKey: String): String {
val key = sessionKey.trim()
if (key.isEmpty()) return appliedMainSessionKey
if (key == "main" && appliedMainSessionKey != "main") return appliedMainSessionKey
return key
}
fun sendMessage(
message: String,
thinkingLevel: String,
attachments: List<OutgoingAttachment>,
) {
scope.launch {
sendMessageAwaitAcceptance(
message = message,
thinkingLevel = thinkingLevel,
attachments = attachments,
)
}
}
suspend fun sendMessageAwaitAcceptance(
message: String,
thinkingLevel: String,
attachments: List<OutgoingAttachment>,
): Boolean {
val trimmed = message.trim()
if (trimmed.isEmpty() && attachments.isEmpty()) return false
if (!_healthOk.value) {
_errorText.value = "Gateway health not OK; cannot send"
return false
}
val runId = UUID.randomUUID().toString()
val text = if (trimmed.isEmpty() && attachments.isNotEmpty()) "See attached." else trimmed
val sessionKey = _sessionKey.value
val thinking = normalizeThinking(thinkingLevel)
// Optimistic user message.
val userContent =
buildList {
add(ChatMessageContent(type = "text", text = text))
for (att in attachments) {
add(
ChatMessageContent(
type = att.type,
mimeType = att.mimeType,
fileName = att.fileName,
base64 = att.base64,
),
)
}
}
_messages.value =
_messages.value +
ChatMessage(
id = UUID.randomUUID().toString(),
role = "user",
content = userContent,
timestampMs = System.currentTimeMillis(),
)
armPendingRunTimeout(runId)
synchronized(pendingRuns) {
pendingRuns.add(runId)
_pendingRunCount.value = pendingRuns.size
}
_errorText.value = null
_streamingAssistantText.value = null
pendingToolCallsById.clear()
publishPendingToolCalls()
return try {
val params =
buildJsonObject {
put("sessionKey", JsonPrimitive(sessionKey))
put("message", JsonPrimitive(text))
put("thinking", JsonPrimitive(thinking))
put("timeoutMs", JsonPrimitive(30_000))
put("idempotencyKey", JsonPrimitive(runId))
if (attachments.isNotEmpty()) {
put(
"attachments",
JsonArray(
attachments.map { att ->
buildJsonObject {
put("type", JsonPrimitive(att.type))
put("mimeType", JsonPrimitive(att.mimeType))
put("fileName", JsonPrimitive(att.fileName))
put("content", JsonPrimitive(att.base64))
}
},
),
)
}
}
val res = session.request("chat.send", params.toString())
val actualRunId = parseRunId(res) ?: runId
if (actualRunId != runId) {
clearPendingRun(runId)
armPendingRunTimeout(actualRunId)
synchronized(pendingRuns) {
pendingRuns.add(actualRunId)
_pendingRunCount.value = pendingRuns.size
}
}
true
} catch (err: Throwable) {
clearPendingRun(runId)
_errorText.value = err.message
false
}
}
fun abort() {
val runIds =
synchronized(pendingRuns) {
pendingRuns.toList()
}
if (runIds.isEmpty()) return
scope.launch {
for (runId in runIds) {
try {
val params =
buildJsonObject {
put("sessionKey", JsonPrimitive(_sessionKey.value))
put("runId", JsonPrimitive(runId))
}
session.request("chat.abort", params.toString())
} catch (_: Throwable) {
// best-effort
}
}
}
}
fun handleGatewayEvent(event: String, payloadJson: String?) {
when (event) {
"tick" -> {
scope.launch { pollHealthIfNeeded(force = false) }
}
"health" -> {
// If we receive a health snapshot, the gateway is reachable.
_healthOk.value = true
}
"seqGap" -> {
_errorText.value = "Event stream interrupted; try refreshing."
clearPendingRuns()
}
"chat" -> {
if (payloadJson.isNullOrBlank()) return
handleChatEvent(payloadJson)
}
"agent" -> {
if (payloadJson.isNullOrBlank()) return
handleAgentEvent(payloadJson)
}
}
}
private suspend fun bootstrap(forceHealth: Boolean, refreshSessions: Boolean) {
_errorText.value = null
_healthOk.value = false
clearPendingRuns()
pendingToolCallsById.clear()
publishPendingToolCalls()
_streamingAssistantText.value = null
_sessionId.value = null
val key = _sessionKey.value
try {
if (supportsChatSubscribe) {
session.sendNodeEvent("chat.subscribe", """{"sessionKey":"$key"}""")
}
val historyJson = session.request("chat.history", """{"sessionKey":"$key"}""")
val history = parseHistory(historyJson, sessionKey = key, previousMessages = _messages.value)
_messages.value = history.messages
_sessionId.value = history.sessionId
history.thinkingLevel?.trim()?.takeIf { it.isNotEmpty() }?.let { _thinkingLevel.value = it }
pollHealthIfNeeded(force = forceHealth)
if (refreshSessions) {
fetchSessions(limit = 50)
}
} catch (err: Throwable) {
_errorText.value = err.message
}
}
private suspend fun fetchSessions(limit: Int?) {
try {
val params =
buildJsonObject {
put("includeGlobal", JsonPrimitive(true))
put("includeUnknown", JsonPrimitive(false))
if (limit != null && limit > 0) put("limit", JsonPrimitive(limit))
}
val res = session.request("sessions.list", params.toString())
_sessions.value = parseSessions(res)
} catch (_: Throwable) {
// best-effort
}
}
private suspend fun pollHealthIfNeeded(force: Boolean) {
val now = System.currentTimeMillis()
val last = lastHealthPollAtMs
if (!force && last != null && now - last < 10_000) return
lastHealthPollAtMs = now
try {
session.request("health", null)
_healthOk.value = true
} catch (_: Throwable) {
_healthOk.value = false
}
}
private fun handleChatEvent(payloadJson: String) {
val payload = json.parseToJsonElement(payloadJson).asObjectOrNull() ?: return
val sessionKey = payload["sessionKey"].asStringOrNull()?.trim()
if (!sessionKey.isNullOrEmpty() && sessionKey != _sessionKey.value) return
val runId = payload["runId"].asStringOrNull()
val isPending =
if (runId != null) synchronized(pendingRuns) { pendingRuns.contains(runId) } else true
val state = payload["state"].asStringOrNull()
when (state) {
"delta" -> {
// Only show streaming text for runs we initiated
if (!isPending) return
val text = parseAssistantDeltaText(payload)
if (!text.isNullOrEmpty()) {
_streamingAssistantText.value = text
}
}
"final", "aborted", "error" -> {
if (state == "error") {
_errorText.value = payload["errorMessage"].asStringOrNull() ?: "Chat failed"
}
if (runId != null) clearPendingRun(runId) else clearPendingRuns()
pendingToolCallsById.clear()
publishPendingToolCalls()
_streamingAssistantText.value = null
scope.launch {
try {
val historyJson =
session.request("chat.history", """{"sessionKey":"${_sessionKey.value}"}""")
val history = parseHistory(historyJson, sessionKey = _sessionKey.value, previousMessages = _messages.value)
_messages.value = history.messages
_sessionId.value = history.sessionId
history.thinkingLevel?.trim()?.takeIf { it.isNotEmpty() }?.let { _thinkingLevel.value = it }
} catch (_: Throwable) {
// best-effort
}
}
}
}
}
private fun handleAgentEvent(payloadJson: String) {
val payload = json.parseToJsonElement(payloadJson).asObjectOrNull() ?: return
val sessionKey = payload["sessionKey"].asStringOrNull()?.trim()
if (!sessionKey.isNullOrEmpty() && sessionKey != _sessionKey.value) return
val stream = payload["stream"].asStringOrNull()
val data = payload["data"].asObjectOrNull()
when (stream) {
"assistant" -> {
val text = data?.get("text")?.asStringOrNull()
if (!text.isNullOrEmpty()) {
_streamingAssistantText.value = text
}
}
"tool" -> {
val phase = data?.get("phase")?.asStringOrNull()
val name = data?.get("name")?.asStringOrNull()
val toolCallId = data?.get("toolCallId")?.asStringOrNull()
if (phase.isNullOrEmpty() || name.isNullOrEmpty() || toolCallId.isNullOrEmpty()) return
val ts = payload["ts"].asLongOrNull() ?: System.currentTimeMillis()
if (phase == "start") {
val args = data?.get("args").asObjectOrNull()
pendingToolCallsById[toolCallId] =
ChatPendingToolCall(
toolCallId = toolCallId,
name = name,
args = args,
startedAtMs = ts,
isError = null,
)
publishPendingToolCalls()
} else if (phase == "result") {
pendingToolCallsById.remove(toolCallId)
publishPendingToolCalls()
}
}
"error" -> {
_errorText.value = "Event stream interrupted; try refreshing."
clearPendingRuns()
pendingToolCallsById.clear()
publishPendingToolCalls()
_streamingAssistantText.value = null
}
}
}
private fun parseAssistantDeltaText(payload: JsonObject): String? {
val message = payload["message"].asObjectOrNull() ?: return null
if (message["role"].asStringOrNull() != "assistant") return null
val content = message["content"].asArrayOrNull() ?: return null
for (item in content) {
val obj = item.asObjectOrNull() ?: continue
if (obj["type"].asStringOrNull() != "text") continue
val text = obj["text"].asStringOrNull()
if (!text.isNullOrEmpty()) {
return text
}
}
return null
}
private fun publishPendingToolCalls() {
_pendingToolCalls.value =
pendingToolCallsById.values.sortedBy { it.startedAtMs }
}
private fun armPendingRunTimeout(runId: String) {
pendingRunTimeoutJobs[runId]?.cancel()
pendingRunTimeoutJobs[runId] =
scope.launch {
delay(pendingRunTimeoutMs)
val stillPending =
synchronized(pendingRuns) {
pendingRuns.contains(runId)
}
if (!stillPending) return@launch
clearPendingRun(runId)
_errorText.value = "Timed out waiting for a reply; try again or refresh."
}
}
private fun clearPendingRun(runId: String) {
pendingRunTimeoutJobs.remove(runId)?.cancel()
synchronized(pendingRuns) {
pendingRuns.remove(runId)
_pendingRunCount.value = pendingRuns.size
}
}
private fun clearPendingRuns() {
for ((_, job) in pendingRunTimeoutJobs) {
job.cancel()
}
pendingRunTimeoutJobs.clear()
synchronized(pendingRuns) {
pendingRuns.clear()
_pendingRunCount.value = 0
}
}
private fun parseHistory(
historyJson: String,
sessionKey: String,
previousMessages: List<ChatMessage>,
): ChatHistory {
val root = json.parseToJsonElement(historyJson).asObjectOrNull() ?: return ChatHistory(sessionKey, null, null, emptyList())
val sid = root["sessionId"].asStringOrNull()
val thinkingLevel = root["thinkingLevel"].asStringOrNull()
val array = root["messages"].asArrayOrNull() ?: JsonArray(emptyList())
val messages =
array.mapNotNull { item ->
val obj = item.asObjectOrNull() ?: return@mapNotNull null
val role = obj["role"].asStringOrNull() ?: return@mapNotNull null
val content = obj["content"].asArrayOrNull()?.mapNotNull(::parseMessageContent) ?: emptyList()
val ts = obj["timestamp"].asLongOrNull()
ChatMessage(
id = UUID.randomUUID().toString(),
role = role,
content = content,
timestampMs = ts,
)
}
return ChatHistory(
sessionKey = sessionKey,
sessionId = sid,
thinkingLevel = thinkingLevel,
messages = reconcileMessageIds(previous = previousMessages, incoming = messages),
)
}
private fun parseMessageContent(el: JsonElement): ChatMessageContent? {
val obj = el.asObjectOrNull() ?: return null
val type = obj["type"].asStringOrNull() ?: "text"
return if (type == "text") {
ChatMessageContent(type = "text", text = obj["text"].asStringOrNull())
} else {
ChatMessageContent(
type = type,
mimeType = obj["mimeType"].asStringOrNull(),
fileName = obj["fileName"].asStringOrNull(),
base64 = obj["content"].asStringOrNull(),
)
}
}
private fun parseSessions(jsonString: String): List<ChatSessionEntry> {
val root = json.parseToJsonElement(jsonString).asObjectOrNull() ?: return emptyList()
val sessions = root["sessions"].asArrayOrNull() ?: return emptyList()
return sessions.mapNotNull { item ->
val obj = item.asObjectOrNull() ?: return@mapNotNull null
val key = obj["key"].asStringOrNull()?.trim().orEmpty()
if (key.isEmpty()) return@mapNotNull null
val updatedAt = obj["updatedAt"].asLongOrNull()
val displayName = obj["displayName"].asStringOrNull()?.trim()
ChatSessionEntry(key = key, updatedAtMs = updatedAt, displayName = displayName)
}
}
private fun parseRunId(resJson: String): String? {
return try {
json.parseToJsonElement(resJson).asObjectOrNull()?.get("runId").asStringOrNull()
} catch (_: Throwable) {
null
}
}
private fun normalizeThinking(raw: String): String {
return when (raw.trim().lowercase()) {
"low" -> "low"
"medium" -> "medium"
"high" -> "high"
else -> "off"
}
}
}
internal data class MainSessionState(
val currentSessionKey: String,
val appliedMainSessionKey: String,
)
internal fun applyMainSessionKey(
currentSessionKey: String,
appliedMainSessionKey: String,
nextMainSessionKey: String,
): MainSessionState {
if (currentSessionKey == appliedMainSessionKey) {
return MainSessionState(
currentSessionKey = nextMainSessionKey,
appliedMainSessionKey = nextMainSessionKey,
)
}
return MainSessionState(
currentSessionKey = currentSessionKey,
appliedMainSessionKey = nextMainSessionKey,
)
}
internal fun reconcileMessageIds(previous: List<ChatMessage>, incoming: List<ChatMessage>): List<ChatMessage> {
if (previous.isEmpty() || incoming.isEmpty()) return incoming
val idsByKey = LinkedHashMap<String, ArrayDeque<String>>()
for (message in previous) {
val key = messageIdentityKey(message) ?: continue
idsByKey.getOrPut(key) { ArrayDeque() }.addLast(message.id)
}
return incoming.map { message ->
val key = messageIdentityKey(message) ?: return@map message
val ids = idsByKey[key] ?: return@map message
val reusedId = ids.removeFirstOrNull() ?: return@map message
if (ids.isEmpty()) {
idsByKey.remove(key)
}
if (reusedId == message.id) return@map message
message.copy(id = reusedId)
}
}
internal fun messageIdentityKey(message: ChatMessage): String? {
val role = message.role.trim().lowercase()
if (role.isEmpty()) return null
val timestamp = message.timestampMs?.toString().orEmpty()
val contentFingerprint =
message.content.joinToString(separator = "\u001E") { part ->
listOf(
part.type.trim().lowercase(),
part.text?.trim().orEmpty(),
part.mimeType?.trim()?.lowercase().orEmpty(),
part.fileName?.trim().orEmpty(),
part.base64?.hashCode()?.toString().orEmpty(),
).joinToString(separator = "\u001F")
}
if (timestamp.isEmpty() && contentFingerprint.isEmpty()) return null
return listOf(role, timestamp, contentFingerprint).joinToString(separator = "|")
}
private fun JsonElement?.asObjectOrNull(): JsonObject? = this as? JsonObject
private fun JsonElement?.asArrayOrNull(): JsonArray? = this as? JsonArray
private fun JsonElement?.asStringOrNull(): String? =
when (this) {
is JsonNull -> null
is JsonPrimitive -> content
else -> null
}
private fun JsonElement?.asLongOrNull(): Long? =
when (this) {
is JsonPrimitive -> content.toLongOrNull()
else -> null
}

View file

@ -0,0 +1,44 @@
package ai.openclaw.app.chat
data class ChatMessage(
val id: String,
val role: String,
val content: List<ChatMessageContent>,
val timestampMs: Long?,
)
data class ChatMessageContent(
val type: String = "text",
val text: String? = null,
val mimeType: String? = null,
val fileName: String? = null,
val base64: String? = null,
)
data class ChatPendingToolCall(
val toolCallId: String,
val name: String,
val args: kotlinx.serialization.json.JsonObject? = null,
val startedAtMs: Long,
val isError: Boolean? = null,
)
data class ChatSessionEntry(
val key: String,
val updatedAtMs: Long?,
val displayName: String? = null,
)
data class ChatHistory(
val sessionKey: String,
val sessionId: String?,
val thinkingLevel: String?,
val messages: List<ChatMessage>,
)
data class OutgoingAttachment(
val type: String,
val mimeType: String,
val fileName: String,
val base64: String,
)

View file

@ -0,0 +1,35 @@
package ai.openclaw.app.gateway
object BonjourEscapes {
fun decode(input: String): String {
if (input.isEmpty()) return input
val bytes = mutableListOf<Byte>()
var i = 0
while (i < input.length) {
if (input[i] == '\\' && i + 3 < input.length) {
val d0 = input[i + 1]
val d1 = input[i + 2]
val d2 = input[i + 3]
if (d0.isDigit() && d1.isDigit() && d2.isDigit()) {
val value =
((d0.code - '0'.code) * 100) + ((d1.code - '0'.code) * 10) + (d2.code - '0'.code)
if (value in 0..255) {
bytes.add(value.toByte())
i += 4
continue
}
}
}
val codePoint = Character.codePointAt(input, i)
val charBytes = String(Character.toChars(codePoint)).toByteArray(Charsets.UTF_8)
for (b in charBytes) {
bytes.add(b)
}
i += Character.charCount(codePoint)
}
return String(bytes.toByteArray(), Charsets.UTF_8)
}
}

View file

@ -0,0 +1,52 @@
package ai.openclaw.app.gateway
internal object DeviceAuthPayload {
fun buildV3(
deviceId: String,
clientId: String,
clientMode: String,
role: String,
scopes: List<String>,
signedAtMs: Long,
token: String?,
nonce: String,
platform: String?,
deviceFamily: String?,
): String {
val scopeString = scopes.joinToString(",")
val authToken = token.orEmpty()
val platformNorm = normalizeMetadataField(platform)
val deviceFamilyNorm = normalizeMetadataField(deviceFamily)
return listOf(
"v3",
deviceId,
clientId,
clientMode,
role,
scopeString,
signedAtMs.toString(),
authToken,
nonce,
platformNorm,
deviceFamilyNorm,
).joinToString("|")
}
internal fun normalizeMetadataField(value: String?): String {
val trimmed = value?.trim().orEmpty()
if (trimmed.isEmpty()) {
return ""
}
// Keep cross-runtime normalization deterministic (TS/Swift/Kotlin):
// lowercase ASCII A-Z only for auth payload metadata fields.
val out = StringBuilder(trimmed.length)
for (ch in trimmed) {
if (ch in 'A'..'Z') {
out.append((ch.code + 32).toChar())
} else {
out.append(ch)
}
}
return out.toString()
}
}

View file

@ -0,0 +1,92 @@
package ai.openclaw.app.gateway
import ai.openclaw.app.SecurePrefs
import kotlinx.serialization.Serializable
import kotlinx.serialization.encodeToString
import kotlinx.serialization.json.Json
data class DeviceAuthEntry(
val token: String,
val role: String,
val scopes: List<String>,
val updatedAtMs: Long,
)
@Serializable
private data class PersistedDeviceAuthMetadata(
val scopes: List<String> = emptyList(),
val updatedAtMs: Long = 0L,
)
interface DeviceAuthTokenStore {
fun loadEntry(deviceId: String, role: String): DeviceAuthEntry?
fun loadToken(deviceId: String, role: String): String? = loadEntry(deviceId, role)?.token
fun saveToken(deviceId: String, role: String, token: String, scopes: List<String> = emptyList())
fun clearToken(deviceId: String, role: String)
}
class DeviceAuthStore(private val prefs: SecurePrefs) : DeviceAuthTokenStore {
private val json = Json { ignoreUnknownKeys = true }
override fun loadEntry(deviceId: String, role: String): DeviceAuthEntry? {
val key = tokenKey(deviceId, role)
val token = prefs.getString(key)?.trim()?.takeIf { it.isNotEmpty() } ?: return null
val normalizedRole = normalizeRole(role)
val metadata =
prefs.getString(metadataKey(deviceId, role))
?.let { raw ->
runCatching { json.decodeFromString<PersistedDeviceAuthMetadata>(raw) }.getOrNull()
}
return DeviceAuthEntry(
token = token,
role = normalizedRole,
scopes = metadata?.scopes ?: emptyList(),
updatedAtMs = metadata?.updatedAtMs ?: 0L,
)
}
override fun saveToken(deviceId: String, role: String, token: String, scopes: List<String>) {
val normalizedScopes = normalizeScopes(scopes)
val key = tokenKey(deviceId, role)
prefs.putString(key, token.trim())
prefs.putString(
metadataKey(deviceId, role),
json.encodeToString(
PersistedDeviceAuthMetadata(
scopes = normalizedScopes,
updatedAtMs = System.currentTimeMillis(),
),
),
)
}
override fun clearToken(deviceId: String, role: String) {
val key = tokenKey(deviceId, role)
prefs.remove(key)
prefs.remove(metadataKey(deviceId, role))
}
private fun tokenKey(deviceId: String, role: String): String {
val normalizedDevice = normalizeDeviceId(deviceId)
val normalizedRole = normalizeRole(role)
return "gateway.deviceToken.$normalizedDevice.$normalizedRole"
}
private fun metadataKey(deviceId: String, role: String): String {
val normalizedDevice = normalizeDeviceId(deviceId)
val normalizedRole = normalizeRole(role)
return "gateway.deviceTokenMeta.$normalizedDevice.$normalizedRole"
}
private fun normalizeDeviceId(deviceId: String): String = deviceId.trim().lowercase()
private fun normalizeRole(role: String): String = role.trim().lowercase()
private fun normalizeScopes(scopes: List<String>): List<String> {
return scopes
.map { it.trim() }
.filter { it.isNotEmpty() }
.distinct()
.sorted()
}
}

View file

@ -0,0 +1,174 @@
package ai.openclaw.app.gateway
import android.content.Context
import android.util.Base64
import java.io.File
import java.security.MessageDigest
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.Json
@Serializable
data class DeviceIdentity(
val deviceId: String,
val publicKeyRawBase64: String,
val privateKeyPkcs8Base64: String,
val createdAtMs: Long,
)
class DeviceIdentityStore(context: Context) {
private val json = Json { ignoreUnknownKeys = true }
private val identityFile = File(context.filesDir, "openclaw/identity/device.json")
@Volatile private var cachedIdentity: DeviceIdentity? = null
@Synchronized
fun loadOrCreate(): DeviceIdentity {
cachedIdentity?.let { return it }
val existing = load()
if (existing != null) {
val derived = deriveDeviceId(existing.publicKeyRawBase64)
if (derived != null && derived != existing.deviceId) {
val updated = existing.copy(deviceId = derived)
save(updated)
cachedIdentity = updated
return updated
}
cachedIdentity = existing
return existing
}
val fresh = generate()
save(fresh)
cachedIdentity = fresh
return fresh
}
fun signPayload(payload: String, identity: DeviceIdentity): String? {
return try {
// Use BC lightweight API directly — JCA provider registration is broken by R8
val privateKeyBytes = Base64.decode(identity.privateKeyPkcs8Base64, Base64.DEFAULT)
val pkInfo = org.bouncycastle.asn1.pkcs.PrivateKeyInfo.getInstance(privateKeyBytes)
val parsed = pkInfo.parsePrivateKey()
val rawPrivate = org.bouncycastle.asn1.DEROctetString.getInstance(parsed).octets
val privateKey = org.bouncycastle.crypto.params.Ed25519PrivateKeyParameters(rawPrivate, 0)
val signer = org.bouncycastle.crypto.signers.Ed25519Signer()
signer.init(true, privateKey)
val payloadBytes = payload.toByteArray(Charsets.UTF_8)
signer.update(payloadBytes, 0, payloadBytes.size)
base64UrlEncode(signer.generateSignature())
} catch (e: Throwable) {
android.util.Log.e("DeviceAuth", "signPayload FAILED: ${e.javaClass.simpleName}: ${e.message}", e)
null
}
}
fun verifySelfSignature(payload: String, signatureBase64Url: String, identity: DeviceIdentity): Boolean {
return try {
val rawPublicKey = Base64.decode(identity.publicKeyRawBase64, Base64.DEFAULT)
val pubKey = org.bouncycastle.crypto.params.Ed25519PublicKeyParameters(rawPublicKey, 0)
val sigBytes = base64UrlDecode(signatureBase64Url)
val verifier = org.bouncycastle.crypto.signers.Ed25519Signer()
verifier.init(false, pubKey)
val payloadBytes = payload.toByteArray(Charsets.UTF_8)
verifier.update(payloadBytes, 0, payloadBytes.size)
verifier.verifySignature(sigBytes)
} catch (e: Throwable) {
android.util.Log.e("DeviceAuth", "self-verify exception: ${e.message}", e)
false
}
}
private fun base64UrlDecode(input: String): ByteArray {
val normalized = input.replace('-', '+').replace('_', '/')
val padded = normalized + "=".repeat((4 - normalized.length % 4) % 4)
return Base64.decode(padded, Base64.DEFAULT)
}
fun publicKeyBase64Url(identity: DeviceIdentity): String? {
return try {
val raw = Base64.decode(identity.publicKeyRawBase64, Base64.DEFAULT)
base64UrlEncode(raw)
} catch (_: Throwable) {
null
}
}
private fun load(): DeviceIdentity? {
return readIdentity(identityFile)
}
private fun readIdentity(file: File): DeviceIdentity? {
return try {
if (!file.exists()) return null
val raw = file.readText(Charsets.UTF_8)
val decoded = json.decodeFromString(DeviceIdentity.serializer(), raw)
if (decoded.deviceId.isBlank() ||
decoded.publicKeyRawBase64.isBlank() ||
decoded.privateKeyPkcs8Base64.isBlank()
) {
null
} else {
decoded
}
} catch (_: Throwable) {
null
}
}
private fun save(identity: DeviceIdentity) {
try {
identityFile.parentFile?.mkdirs()
val encoded = json.encodeToString(DeviceIdentity.serializer(), identity)
identityFile.writeText(encoded, Charsets.UTF_8)
} catch (_: Throwable) {
// best-effort only
}
}
private fun generate(): DeviceIdentity {
// Use BC lightweight API directly to avoid JCA provider issues with R8
val kpGen = org.bouncycastle.crypto.generators.Ed25519KeyPairGenerator()
kpGen.init(org.bouncycastle.crypto.params.Ed25519KeyGenerationParameters(java.security.SecureRandom()))
val kp = kpGen.generateKeyPair()
val pubKey = kp.public as org.bouncycastle.crypto.params.Ed25519PublicKeyParameters
val privKey = kp.private as org.bouncycastle.crypto.params.Ed25519PrivateKeyParameters
val rawPublic = pubKey.encoded // 32 bytes
val deviceId = sha256Hex(rawPublic)
// Encode private key as PKCS8 for storage
val privKeyInfo = org.bouncycastle.crypto.util.PrivateKeyInfoFactory.createPrivateKeyInfo(privKey)
val pkcs8Bytes = privKeyInfo.encoded
return DeviceIdentity(
deviceId = deviceId,
publicKeyRawBase64 = Base64.encodeToString(rawPublic, Base64.NO_WRAP),
privateKeyPkcs8Base64 = Base64.encodeToString(pkcs8Bytes, Base64.NO_WRAP),
createdAtMs = System.currentTimeMillis(),
)
}
private fun deriveDeviceId(publicKeyRawBase64: String): String? {
return try {
val raw = Base64.decode(publicKeyRawBase64, Base64.DEFAULT)
sha256Hex(raw)
} catch (_: Throwable) {
null
}
}
private fun sha256Hex(data: ByteArray): String {
val digest = MessageDigest.getInstance("SHA-256").digest(data)
val out = CharArray(digest.size * 2)
var i = 0
for (byte in digest) {
val v = byte.toInt() and 0xff
out[i++] = HEX[v ushr 4]
out[i++] = HEX[v and 0x0f]
}
return String(out)
}
private fun base64UrlEncode(data: ByteArray): String {
return Base64.encodeToString(data, Base64.URL_SAFE or Base64.NO_WRAP or Base64.NO_PADDING)
}
companion object {
private val HEX = "0123456789abcdef".toCharArray()
}
}

View file

@ -0,0 +1,522 @@
package ai.openclaw.app.gateway
import android.content.Context
import android.net.ConnectivityManager
import android.net.DnsResolver
import android.net.NetworkCapabilities
import android.net.nsd.NsdManager
import android.net.nsd.NsdServiceInfo
import android.os.CancellationSignal
import android.util.Log
import java.io.IOException
import java.net.InetSocketAddress
import java.nio.ByteBuffer
import java.nio.charset.CodingErrorAction
import java.time.Duration
import java.util.concurrent.ConcurrentHashMap
import java.util.concurrent.Executor
import java.util.concurrent.Executors
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.Job
import kotlinx.coroutines.delay
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asStateFlow
import kotlinx.coroutines.launch
import kotlinx.coroutines.suspendCancellableCoroutine
import org.xbill.DNS.AAAARecord
import org.xbill.DNS.ARecord
import org.xbill.DNS.DClass
import org.xbill.DNS.ExtendedResolver
import org.xbill.DNS.Message
import org.xbill.DNS.Name
import org.xbill.DNS.PTRRecord
import org.xbill.DNS.Record
import org.xbill.DNS.Rcode
import org.xbill.DNS.Resolver
import org.xbill.DNS.SRVRecord
import org.xbill.DNS.Section
import org.xbill.DNS.SimpleResolver
import org.xbill.DNS.TextParseException
import org.xbill.DNS.TXTRecord
import org.xbill.DNS.Type
import kotlin.coroutines.resume
import kotlin.coroutines.resumeWithException
@Suppress("DEPRECATION")
class GatewayDiscovery(
context: Context,
private val scope: CoroutineScope,
) {
private val nsd = context.getSystemService(NsdManager::class.java)
private val connectivity = context.getSystemService(ConnectivityManager::class.java)
private val dns = DnsResolver.getInstance()
private val serviceType = "_openclaw-gw._tcp."
private val wideAreaDomain = System.getenv("OPENCLAW_WIDE_AREA_DOMAIN")
private val logTag = "OpenClaw/GatewayDiscovery"
private val localById = ConcurrentHashMap<String, GatewayEndpoint>()
private val unicastById = ConcurrentHashMap<String, GatewayEndpoint>()
private val _gateways = MutableStateFlow<List<GatewayEndpoint>>(emptyList())
val gateways: StateFlow<List<GatewayEndpoint>> = _gateways.asStateFlow()
private val _statusText = MutableStateFlow("Searching…")
val statusText: StateFlow<String> = _statusText.asStateFlow()
private var unicastJob: Job? = null
private val dnsExecutor: Executor = Executors.newCachedThreadPool()
@Volatile private var lastWideAreaRcode: Int? = null
@Volatile private var lastWideAreaCount: Int = 0
private val discoveryListener =
object : NsdManager.DiscoveryListener {
override fun onStartDiscoveryFailed(serviceType: String, errorCode: Int) {}
override fun onStopDiscoveryFailed(serviceType: String, errorCode: Int) {}
override fun onDiscoveryStarted(serviceType: String) {}
override fun onDiscoveryStopped(serviceType: String) {}
override fun onServiceFound(serviceInfo: NsdServiceInfo) {
if (serviceInfo.serviceType != this@GatewayDiscovery.serviceType) return
resolve(serviceInfo)
}
override fun onServiceLost(serviceInfo: NsdServiceInfo) {
val serviceName = BonjourEscapes.decode(serviceInfo.serviceName)
val id = stableId(serviceName, "local.")
localById.remove(id)
publish()
}
}
init {
startLocalDiscovery()
if (!wideAreaDomain.isNullOrBlank()) {
startUnicastDiscovery(wideAreaDomain)
}
}
private fun startLocalDiscovery() {
try {
nsd.discoverServices(serviceType, NsdManager.PROTOCOL_DNS_SD, discoveryListener)
} catch (_: Throwable) {
// ignore (best-effort)
}
}
private fun stopLocalDiscovery() {
try {
nsd.stopServiceDiscovery(discoveryListener)
} catch (_: Throwable) {
// ignore (best-effort)
}
}
private fun startUnicastDiscovery(domain: String) {
unicastJob =
scope.launch(Dispatchers.IO) {
while (true) {
try {
refreshUnicast(domain)
} catch (_: Throwable) {
// ignore (best-effort)
}
delay(5000)
}
}
}
private fun resolve(serviceInfo: NsdServiceInfo) {
nsd.resolveService(
serviceInfo,
object : NsdManager.ResolveListener {
override fun onResolveFailed(serviceInfo: NsdServiceInfo, errorCode: Int) {}
override fun onServiceResolved(resolved: NsdServiceInfo) {
val host = resolved.host?.hostAddress ?: return
val port = resolved.port
if (port <= 0) return
val rawServiceName = resolved.serviceName
val serviceName = BonjourEscapes.decode(rawServiceName)
val displayName = BonjourEscapes.decode(txt(resolved, "displayName") ?: serviceName)
val lanHost = txt(resolved, "lanHost")
val tailnetDns = txt(resolved, "tailnetDns")
val gatewayPort = txtInt(resolved, "gatewayPort")
val canvasPort = txtInt(resolved, "canvasPort")
val tlsEnabled = txtBool(resolved, "gatewayTls")
val tlsFingerprint = txt(resolved, "gatewayTlsSha256")
val id = stableId(serviceName, "local.")
localById[id] =
GatewayEndpoint(
stableId = id,
name = displayName,
host = host,
port = port,
lanHost = lanHost,
tailnetDns = tailnetDns,
gatewayPort = gatewayPort,
canvasPort = canvasPort,
tlsEnabled = tlsEnabled,
tlsFingerprintSha256 = tlsFingerprint,
)
publish()
}
},
)
}
private fun publish() {
_gateways.value =
(localById.values + unicastById.values).sortedBy { it.name.lowercase() }
_statusText.value = buildStatusText()
}
private fun buildStatusText(): String {
val localCount = localById.size
val wideRcode = lastWideAreaRcode
val wideCount = lastWideAreaCount
val wide =
when (wideRcode) {
null -> "Wide: ?"
Rcode.NOERROR -> "Wide: $wideCount"
Rcode.NXDOMAIN -> "Wide: NXDOMAIN"
else -> "Wide: ${Rcode.string(wideRcode)}"
}
return when {
localCount == 0 && wideRcode == null -> "Searching for gateways…"
localCount == 0 -> "$wide"
else -> "Local: $localCount • $wide"
}
}
private fun stableId(serviceName: String, domain: String): String {
return "${serviceType}|${domain}|${normalizeName(serviceName)}"
}
private fun normalizeName(raw: String): String {
return raw.trim().split(Regex("\\s+")).joinToString(" ")
}
private fun txt(info: NsdServiceInfo, key: String): String? {
val bytes = info.attributes[key] ?: return null
return try {
String(bytes, Charsets.UTF_8).trim().ifEmpty { null }
} catch (_: Throwable) {
null
}
}
private fun txtInt(info: NsdServiceInfo, key: String): Int? {
return txt(info, key)?.toIntOrNull()
}
private fun txtBool(info: NsdServiceInfo, key: String): Boolean {
val raw = txt(info, key)?.trim()?.lowercase() ?: return false
return raw == "1" || raw == "true" || raw == "yes"
}
private suspend fun refreshUnicast(domain: String) {
val ptrName = "${serviceType}${domain}"
val ptrMsg = lookupUnicastMessage(ptrName, Type.PTR) ?: return
val ptrRecords = records(ptrMsg, Section.ANSWER).mapNotNull { it as? PTRRecord }
val next = LinkedHashMap<String, GatewayEndpoint>()
for (ptr in ptrRecords) {
val instanceFqdn = ptr.target.toString()
val srv =
recordByName(ptrMsg, instanceFqdn, Type.SRV) as? SRVRecord
?: run {
val msg = lookupUnicastMessage(instanceFqdn, Type.SRV) ?: return@run null
recordByName(msg, instanceFqdn, Type.SRV) as? SRVRecord
}
?: continue
val port = srv.port
if (port <= 0) continue
val targetFqdn = srv.target.toString()
val host =
resolveHostFromMessage(ptrMsg, targetFqdn)
?: resolveHostFromMessage(lookupUnicastMessage(instanceFqdn, Type.SRV), targetFqdn)
?: resolveHostUnicast(targetFqdn)
?: continue
val txtFromPtr =
recordsByName(ptrMsg, Section.ADDITIONAL)[keyName(instanceFqdn)]
.orEmpty()
.mapNotNull { it as? TXTRecord }
val txt =
if (txtFromPtr.isNotEmpty()) {
txtFromPtr
} else {
val msg = lookupUnicastMessage(instanceFqdn, Type.TXT)
records(msg, Section.ANSWER).mapNotNull { it as? TXTRecord }
}
val instanceName = BonjourEscapes.decode(decodeInstanceName(instanceFqdn, domain))
val displayName = BonjourEscapes.decode(txtValue(txt, "displayName") ?: instanceName)
val lanHost = txtValue(txt, "lanHost")
val tailnetDns = txtValue(txt, "tailnetDns")
val gatewayPort = txtIntValue(txt, "gatewayPort")
val canvasPort = txtIntValue(txt, "canvasPort")
val tlsEnabled = txtBoolValue(txt, "gatewayTls")
val tlsFingerprint = txtValue(txt, "gatewayTlsSha256")
val id = stableId(instanceName, domain)
next[id] =
GatewayEndpoint(
stableId = id,
name = displayName,
host = host,
port = port,
lanHost = lanHost,
tailnetDns = tailnetDns,
gatewayPort = gatewayPort,
canvasPort = canvasPort,
tlsEnabled = tlsEnabled,
tlsFingerprintSha256 = tlsFingerprint,
)
}
unicastById.clear()
unicastById.putAll(next)
lastWideAreaRcode = ptrMsg.header.rcode
lastWideAreaCount = next.size
publish()
if (next.isEmpty()) {
Log.d(
logTag,
"wide-area discovery: 0 results for $ptrName (rcode=${Rcode.string(ptrMsg.header.rcode)})",
)
}
}
private fun decodeInstanceName(instanceFqdn: String, domain: String): String {
val suffix = "${serviceType}${domain}"
val withoutSuffix =
if (instanceFqdn.endsWith(suffix)) {
instanceFqdn.removeSuffix(suffix)
} else {
instanceFqdn.substringBefore(serviceType)
}
return normalizeName(stripTrailingDot(withoutSuffix))
}
private fun stripTrailingDot(raw: String): String {
return raw.removeSuffix(".")
}
private suspend fun lookupUnicastMessage(name: String, type: Int): Message? {
val query =
try {
Message.newQuery(
org.xbill.DNS.Record.newRecord(
Name.fromString(name),
type,
DClass.IN,
),
)
} catch (_: TextParseException) {
return null
}
val system = queryViaSystemDns(query)
if (records(system, Section.ANSWER).any { it.type == type }) return system
val direct = createDirectResolver() ?: return system
return try {
val msg = direct.send(query)
if (records(msg, Section.ANSWER).any { it.type == type }) msg else system
} catch (_: Throwable) {
system
}
}
private suspend fun queryViaSystemDns(query: Message): Message? {
val network = preferredDnsNetwork()
val bytes =
try {
rawQuery(network, query.toWire())
} catch (_: Throwable) {
return null
}
return try {
Message(bytes)
} catch (_: IOException) {
null
}
}
private fun records(msg: Message?, section: Int): List<Record> {
return msg?.getSection(section).orEmpty()
}
private fun keyName(raw: String): String {
return raw.trim().lowercase()
}
private fun recordsByName(msg: Message, section: Int): Map<String, List<Record>> {
val next = LinkedHashMap<String, MutableList<Record>>()
for (r in records(msg, section)) {
val name = r.name?.toString() ?: continue
next.getOrPut(keyName(name)) { mutableListOf() }.add(r)
}
return next
}
private fun recordByName(msg: Message, fqdn: String, type: Int): Record? {
val key = keyName(fqdn)
val byNameAnswer = recordsByName(msg, Section.ANSWER)
val fromAnswer = byNameAnswer[key].orEmpty().firstOrNull { it.type == type }
if (fromAnswer != null) return fromAnswer
val byNameAdditional = recordsByName(msg, Section.ADDITIONAL)
return byNameAdditional[key].orEmpty().firstOrNull { it.type == type }
}
private fun resolveHostFromMessage(msg: Message?, hostname: String): String? {
val m = msg ?: return null
val key = keyName(hostname)
val additional = recordsByName(m, Section.ADDITIONAL)[key].orEmpty()
val a = additional.mapNotNull { it as? ARecord }.mapNotNull { it.address?.hostAddress }
val aaaa = additional.mapNotNull { it as? AAAARecord }.mapNotNull { it.address?.hostAddress }
return a.firstOrNull() ?: aaaa.firstOrNull()
}
private fun preferredDnsNetwork(): android.net.Network? {
val cm = connectivity ?: return null
// Prefer VPN (Tailscale) when present; otherwise use the active network.
cm.allNetworks.firstOrNull { n ->
val caps = cm.getNetworkCapabilities(n) ?: return@firstOrNull false
caps.hasTransport(NetworkCapabilities.TRANSPORT_VPN)
}?.let { return it }
return cm.activeNetwork
}
private fun createDirectResolver(): Resolver? {
val cm = connectivity ?: return null
val candidateNetworks =
buildList {
cm.allNetworks
.firstOrNull { n ->
val caps = cm.getNetworkCapabilities(n) ?: return@firstOrNull false
caps.hasTransport(NetworkCapabilities.TRANSPORT_VPN)
}?.let(::add)
cm.activeNetwork?.let(::add)
}.distinct()
val servers =
candidateNetworks
.asSequence()
.flatMap { n ->
cm.getLinkProperties(n)?.dnsServers?.asSequence() ?: emptySequence()
}
.distinctBy { it.hostAddress ?: it.toString() }
.toList()
if (servers.isEmpty()) return null
return try {
val resolvers =
servers.mapNotNull { addr ->
try {
SimpleResolver().apply {
setAddress(InetSocketAddress(addr, 53))
setTimeout(Duration.ofSeconds(3))
}
} catch (_: Throwable) {
null
}
}
if (resolvers.isEmpty()) return null
ExtendedResolver(resolvers.toTypedArray()).apply { setTimeout(Duration.ofSeconds(3)) }
} catch (_: Throwable) {
null
}
}
private suspend fun rawQuery(network: android.net.Network?, wireQuery: ByteArray): ByteArray =
suspendCancellableCoroutine { cont ->
val signal = CancellationSignal()
cont.invokeOnCancellation { signal.cancel() }
dns.rawQuery(
network,
wireQuery,
DnsResolver.FLAG_EMPTY,
dnsExecutor,
signal,
object : DnsResolver.Callback<ByteArray> {
override fun onAnswer(answer: ByteArray, rcode: Int) {
cont.resume(answer)
}
override fun onError(error: DnsResolver.DnsException) {
cont.resumeWithException(error)
}
},
)
}
private fun txtValue(records: List<TXTRecord>, key: String): String? {
val prefix = "$key="
for (r in records) {
val strings: List<String> =
try {
r.strings.mapNotNull { it as? String }
} catch (_: Throwable) {
emptyList()
}
for (s in strings) {
val trimmed = decodeDnsTxtString(s).trim()
if (trimmed.startsWith(prefix)) {
return trimmed.removePrefix(prefix).trim().ifEmpty { null }
}
}
}
return null
}
private fun txtIntValue(records: List<TXTRecord>, key: String): Int? {
return txtValue(records, key)?.toIntOrNull()
}
private fun txtBoolValue(records: List<TXTRecord>, key: String): Boolean {
val raw = txtValue(records, key)?.trim()?.lowercase() ?: return false
return raw == "1" || raw == "true" || raw == "yes"
}
private fun decodeDnsTxtString(raw: String): String {
// dnsjava treats TXT as opaque bytes and decodes as ISO-8859-1 to preserve bytes.
// Our TXT payload is UTF-8 (written by the gateway), so re-decode when possible.
val bytes = raw.toByteArray(Charsets.ISO_8859_1)
val decoder =
Charsets.UTF_8
.newDecoder()
.onMalformedInput(CodingErrorAction.REPORT)
.onUnmappableCharacter(CodingErrorAction.REPORT)
return try {
decoder.decode(ByteBuffer.wrap(bytes)).toString()
} catch (_: Throwable) {
raw
}
}
private suspend fun resolveHostUnicast(hostname: String): String? {
val a =
records(lookupUnicastMessage(hostname, Type.A), Section.ANSWER)
.mapNotNull { it as? ARecord }
.mapNotNull { it.address?.hostAddress }
val aaaa =
records(lookupUnicastMessage(hostname, Type.AAAA), Section.ANSWER)
.mapNotNull { it as? AAAARecord }
.mapNotNull { it.address?.hostAddress }
return a.firstOrNull() ?: aaaa.firstOrNull()
}
}

View file

@ -0,0 +1,26 @@
package ai.openclaw.app.gateway
data class GatewayEndpoint(
val stableId: String,
val name: String,
val host: String,
val port: Int,
val lanHost: String? = null,
val tailnetDns: String? = null,
val gatewayPort: Int? = null,
val canvasPort: Int? = null,
val tlsEnabled: Boolean = false,
val tlsFingerprintSha256: String? = null,
) {
companion object {
fun manual(host: String, port: Int): GatewayEndpoint =
GatewayEndpoint(
stableId = "manual|${host.lowercase()}|$port",
name = "$host:$port",
host = host,
port = port,
tlsEnabled = false,
tlsFingerprintSha256 = null,
)
}
}

View file

@ -0,0 +1,124 @@
package ai.openclaw.app.gateway
import android.os.Build
import java.net.InetAddress
import java.util.Locale
internal fun isLoopbackGatewayHost(
rawHost: String?,
allowEmulatorBridgeAlias: Boolean = isAndroidEmulatorRuntime(),
): Boolean {
var host =
rawHost
?.trim()
?.lowercase(Locale.US)
?.trim('[', ']')
.orEmpty()
if (host.endsWith(".")) {
host = host.dropLast(1)
}
val zoneIndex = host.indexOf('%')
if (zoneIndex >= 0) return false
if (host.isEmpty()) return false
if (host == "localhost") return true
if (allowEmulatorBridgeAlias && host == "10.0.2.2") return true
parseIpv4Address(host)?.let { ipv4 ->
return ipv4.first() == 127.toByte()
}
if (!host.contains(':') || !host.all(::isIpv6LiteralChar)) return false
val address = runCatching { InetAddress.getByName(host) }.getOrNull()?.address ?: return false
if (address.size == 4) {
return address[0] == 127.toByte()
}
if (address.size != 16) return false
// `::1` is 15 zero bytes followed by `0x01`.
val isIpv6Loopback = address.copyOfRange(0, 15).all { it == 0.toByte() } && address[15] == 1.toByte()
if (isIpv6Loopback) return true
val isMappedIpv4 =
address.copyOfRange(0, 10).all { it == 0.toByte() } &&
address[10] == 0xFF.toByte() &&
address[11] == 0xFF.toByte()
return isMappedIpv4 && address[12] == 127.toByte()
}
internal fun isPrivateLanGatewayHost(
rawHost: String?,
allowEmulatorBridgeAlias: Boolean = isAndroidEmulatorRuntime(),
): Boolean {
var host =
rawHost
?.trim()
?.lowercase(Locale.US)
?.trim('[', ']')
.orEmpty()
if (host.endsWith(".")) {
host = host.dropLast(1)
}
val zoneIndex = host.indexOf('%')
if (zoneIndex >= 0) {
host = host.substring(0, zoneIndex)
}
if (host.isEmpty()) return false
if (isLoopbackGatewayHost(host, allowEmulatorBridgeAlias = allowEmulatorBridgeAlias)) return true
if (host.endsWith(".local")) return true
if (!host.contains('.') && !host.contains(':')) return true
parseIpv4Address(host)?.let { ipv4 ->
val first = ipv4[0].toInt() and 0xff
val second = ipv4[1].toInt() and 0xff
return when {
first == 10 -> true
first == 172 && second in 16..31 -> true
first == 192 && second == 168 -> true
first == 169 && second == 254 -> true
else -> false
}
}
if (!host.contains(':') || !host.all(::isIpv6LiteralChar)) return false
val address = runCatching { InetAddress.getByName(host) }.getOrNull() ?: return false
return when {
address.isLinkLocalAddress -> true
address.isSiteLocalAddress -> true
else -> {
val bytes = address.address
bytes.size == 16 && (bytes[0].toInt() and 0xfe) == 0xfc
}
}
}
private fun isAndroidEmulatorRuntime(): Boolean {
val fingerprint = Build.FINGERPRINT?.lowercase(Locale.US).orEmpty()
val model = Build.MODEL?.lowercase(Locale.US).orEmpty()
val manufacturer = Build.MANUFACTURER?.lowercase(Locale.US).orEmpty()
val brand = Build.BRAND?.lowercase(Locale.US).orEmpty()
val device = Build.DEVICE?.lowercase(Locale.US).orEmpty()
val product = Build.PRODUCT?.lowercase(Locale.US).orEmpty()
return fingerprint.contains("generic") ||
fingerprint.contains("robolectric") ||
model.contains("emulator") ||
model.contains("sdk_gphone") ||
manufacturer.contains("genymotion") ||
(brand.contains("generic") && device.contains("generic")) ||
product.contains("sdk_gphone") ||
product.contains("emulator") ||
product.contains("simulator")
}
private fun parseIpv4Address(host: String): ByteArray? {
val parts = host.split('.')
if (parts.size != 4) return null
val bytes = ByteArray(4)
for ((index, part) in parts.withIndex()) {
val value = part.toIntOrNull() ?: return null
if (value !in 0..255) return null
bytes[index] = value.toByte()
}
return bytes
}
private fun isIpv6LiteralChar(char: Char): Boolean = char in '0'..'9' || char in 'a'..'f' || char == ':' || char == '.'

View file

@ -0,0 +1,3 @@
package ai.openclaw.app.gateway
const val GATEWAY_PROTOCOL_VERSION = 3

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,185 @@
package ai.openclaw.app.gateway
import android.annotation.SuppressLint
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import java.io.EOFException
import java.net.ConnectException
import java.net.InetSocketAddress
import java.net.SocketTimeoutException
import java.net.UnknownHostException
import java.security.MessageDigest
import java.security.SecureRandom
import java.security.cert.CertificateException
import java.security.cert.X509Certificate
import java.util.Locale
import javax.net.ssl.HttpsURLConnection
import javax.net.ssl.HostnameVerifier
import javax.net.ssl.SSLContext
import javax.net.ssl.SSLException
import javax.net.ssl.SSLParameters
import javax.net.ssl.SSLSocketFactory
import javax.net.ssl.SNIHostName
import javax.net.ssl.SSLSocket
import javax.net.ssl.TrustManagerFactory
import javax.net.ssl.X509TrustManager
data class GatewayTlsParams(
val required: Boolean,
val expectedFingerprint: String?,
val allowTOFU: Boolean,
val stableId: String,
)
data class GatewayTlsConfig(
val sslSocketFactory: SSLSocketFactory,
val trustManager: X509TrustManager,
val hostnameVerifier: HostnameVerifier,
)
enum class GatewayTlsProbeFailure {
TLS_UNAVAILABLE,
ENDPOINT_UNREACHABLE,
}
data class GatewayTlsProbeResult(
val fingerprintSha256: String? = null,
val failure: GatewayTlsProbeFailure? = null,
)
fun buildGatewayTlsConfig(
params: GatewayTlsParams?,
onStore: ((String) -> Unit)? = null,
): GatewayTlsConfig? {
if (params == null) return null
val expected = params.expectedFingerprint?.let(::normalizeFingerprint)
val defaultTrust = defaultTrustManager()
@SuppressLint("CustomX509TrustManager")
val trustManager =
object : X509TrustManager {
override fun checkClientTrusted(chain: Array<X509Certificate>, authType: String) {
defaultTrust.checkClientTrusted(chain, authType)
}
override fun checkServerTrusted(chain: Array<X509Certificate>, authType: String) {
if (chain.isEmpty()) throw CertificateException("empty certificate chain")
val fingerprint = sha256Hex(chain[0].encoded)
if (expected != null) {
if (fingerprint != expected) {
throw CertificateException("gateway TLS fingerprint mismatch")
}
return
}
if (params.allowTOFU) {
onStore?.invoke(fingerprint)
return
}
defaultTrust.checkServerTrusted(chain, authType)
}
override fun getAcceptedIssuers(): Array<X509Certificate> = defaultTrust.acceptedIssuers
}
val context = SSLContext.getInstance("TLS")
context.init(null, arrayOf(trustManager), SecureRandom())
val verifier =
if (expected != null || params.allowTOFU) {
// When pinning, we intentionally ignore hostname mismatch (service discovery often yields IPs).
HostnameVerifier { _, _ -> true }
} else {
HttpsURLConnection.getDefaultHostnameVerifier()
}
return GatewayTlsConfig(
sslSocketFactory = context.socketFactory,
trustManager = trustManager,
hostnameVerifier = verifier,
)
}
suspend fun probeGatewayTlsFingerprint(
host: String,
port: Int,
timeoutMs: Int = 3_000,
): GatewayTlsProbeResult {
val trimmedHost = host.trim()
if (trimmedHost.isEmpty()) return GatewayTlsProbeResult(failure = GatewayTlsProbeFailure.ENDPOINT_UNREACHABLE)
if (port !in 1..65535) return GatewayTlsProbeResult(failure = GatewayTlsProbeFailure.ENDPOINT_UNREACHABLE)
return withContext(Dispatchers.IO) {
val trustAll =
@SuppressLint("CustomX509TrustManager", "TrustAllX509TrustManager")
object : X509TrustManager {
@SuppressLint("TrustAllX509TrustManager")
override fun checkClientTrusted(chain: Array<X509Certificate>, authType: String) {}
@SuppressLint("TrustAllX509TrustManager")
override fun checkServerTrusted(chain: Array<X509Certificate>, authType: String) {}
override fun getAcceptedIssuers(): Array<X509Certificate> = emptyArray()
}
val context = SSLContext.getInstance("TLS")
context.init(null, arrayOf(trustAll), SecureRandom())
val socket = (context.socketFactory.createSocket() as SSLSocket)
try {
socket.soTimeout = timeoutMs
socket.connect(InetSocketAddress(trimmedHost, port), timeoutMs)
// Best-effort SNI for hostnames (avoid crashing on IP literals).
try {
if (trimmedHost.any { it.isLetter() }) {
val params = SSLParameters()
params.serverNames = listOf(SNIHostName(trimmedHost))
socket.sslParameters = params
}
} catch (_: Throwable) {
// ignore
}
socket.startHandshake()
val cert =
socket.session.peerCertificates.firstOrNull() as? X509Certificate
?: return@withContext GatewayTlsProbeResult(failure = GatewayTlsProbeFailure.TLS_UNAVAILABLE)
GatewayTlsProbeResult(fingerprintSha256 = sha256Hex(cert.encoded))
} catch (err: Throwable) {
val failure =
when (err) {
is SSLException,
is EOFException -> GatewayTlsProbeFailure.TLS_UNAVAILABLE
is ConnectException,
is SocketTimeoutException,
is UnknownHostException -> GatewayTlsProbeFailure.ENDPOINT_UNREACHABLE
else -> GatewayTlsProbeFailure.ENDPOINT_UNREACHABLE
}
GatewayTlsProbeResult(failure = failure)
} finally {
try {
socket.close()
} catch (_: Throwable) {
// ignore
}
}
}
}
private fun defaultTrustManager(): X509TrustManager {
val factory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm())
factory.init(null as java.security.KeyStore?)
val trust =
factory.trustManagers.firstOrNull { it is X509TrustManager } as? X509TrustManager
return trust ?: throw IllegalStateException("No default X509TrustManager found")
}
private fun sha256Hex(data: ByteArray): String {
val digest = MessageDigest.getInstance("SHA-256").digest(data)
val out = StringBuilder(digest.size * 2)
for (byte in digest) {
out.append(String.format(Locale.US, "%02x", byte))
}
return out.toString()
}
private fun normalizeFingerprint(raw: String): String {
val stripped = raw.trim()
.replace(Regex("^sha-?256\\s*:?\\s*", RegexOption.IGNORE_CASE), "")
return stripped.lowercase(Locale.US).filter { it in '0'..'9' || it in 'a'..'f' }
}

View file

@ -0,0 +1,39 @@
package ai.openclaw.app.gateway
data class ParsedInvokeError(
val code: String,
val message: String,
val hadExplicitCode: Boolean,
) {
val prefixedMessage: String
get() = "$code: $message"
}
fun parseInvokeErrorMessage(raw: String): ParsedInvokeError {
val trimmed = raw.trim()
if (trimmed.isEmpty()) {
return ParsedInvokeError(code = "UNAVAILABLE", message = "error", hadExplicitCode = false)
}
val parts = trimmed.split(":", limit = 2)
if (parts.size == 2) {
val code = parts[0].trim()
val rest = parts[1].trim()
if (code.isNotEmpty() && code.all { it.isUpperCase() || it == '_' }) {
return ParsedInvokeError(
code = code,
message = rest.ifEmpty { trimmed },
hadExplicitCode = true,
)
}
}
return ParsedInvokeError(code = "UNAVAILABLE", message = trimmed, hadExplicitCode = false)
}
fun parseInvokeErrorFromThrowable(
err: Throwable,
fallbackMessage: String = "error",
): ParsedInvokeError {
val raw = err.message?.trim().takeIf { !it.isNullOrEmpty() } ?: fallbackMessage
return parseInvokeErrorMessage(raw)
}

View file

@ -0,0 +1,153 @@
package ai.openclaw.app.node
import ai.openclaw.app.gateway.GatewaySession
import kotlinx.coroutines.delay
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonArray
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
class A2UIHandler(
private val canvas: CanvasController,
private val json: Json,
private val getNodeCanvasHostUrl: () -> String?,
private val getOperatorCanvasHostUrl: () -> String?,
) {
fun isTrustedCanvasActionUrl(rawUrl: String?): Boolean {
return CanvasActionTrust.isTrustedCanvasActionUrl(
rawUrl = rawUrl,
trustedA2uiUrls = listOfNotNull(resolveA2uiHostUrl()),
)
}
fun resolveA2uiHostUrl(): String? {
val nodeRaw = getNodeCanvasHostUrl()?.trim().orEmpty()
val operatorRaw = getOperatorCanvasHostUrl()?.trim().orEmpty()
val raw = if (nodeRaw.isNotBlank()) nodeRaw else operatorRaw
if (raw.isBlank()) return null
val base = raw.trimEnd('/')
return "${base}/__openclaw__/a2ui/?platform=android"
}
suspend fun ensureA2uiReady(a2uiUrl: String): Boolean {
try {
val already = canvas.eval(a2uiReadyCheckJS)
if (already == "true") return true
} catch (_: Throwable) {
// ignore
}
canvas.navigate(a2uiUrl)
repeat(50) {
try {
val ready = canvas.eval(a2uiReadyCheckJS)
if (ready == "true") return true
} catch (_: Throwable) {
// ignore
}
delay(120)
}
return false
}
fun decodeA2uiMessages(command: String, paramsJson: String?): String {
val raw = paramsJson?.trim().orEmpty()
if (raw.isBlank()) throw IllegalArgumentException("INVALID_REQUEST: paramsJSON required")
val obj =
json.parseToJsonElement(raw) as? JsonObject
?: throw IllegalArgumentException("INVALID_REQUEST: expected object params")
val jsonlField = (obj["jsonl"] as? JsonPrimitive)?.content?.trim().orEmpty()
val hasMessagesArray = obj["messages"] is JsonArray
if (command == "canvas.a2ui.pushJSONL" || (!hasMessagesArray && jsonlField.isNotBlank())) {
val jsonl = jsonlField
if (jsonl.isBlank()) throw IllegalArgumentException("INVALID_REQUEST: jsonl required")
val messages =
jsonl
.lineSequence()
.map { it.trim() }
.filter { it.isNotBlank() }
.mapIndexed { idx, line ->
val el = json.parseToJsonElement(line)
val msg =
el as? JsonObject
?: throw IllegalArgumentException("A2UI JSONL line ${idx + 1}: expected a JSON object")
validateA2uiV0_8(msg, idx + 1)
msg
}
.toList()
return JsonArray(messages).toString()
}
val arr = obj["messages"] as? JsonArray ?: throw IllegalArgumentException("INVALID_REQUEST: messages[] required")
val out =
arr.mapIndexed { idx, el ->
val msg =
el as? JsonObject
?: throw IllegalArgumentException("A2UI messages[${idx}]: expected a JSON object")
validateA2uiV0_8(msg, idx + 1)
msg
}
return JsonArray(out).toString()
}
private fun validateA2uiV0_8(msg: JsonObject, lineNumber: Int) {
if (msg.containsKey("createSurface")) {
throw IllegalArgumentException(
"A2UI JSONL line $lineNumber: looks like A2UI v0.9 (`createSurface`). Canvas supports v0.8 messages only.",
)
}
val allowed = setOf("beginRendering", "surfaceUpdate", "dataModelUpdate", "deleteSurface")
val matched = msg.keys.filter { allowed.contains(it) }
if (matched.size != 1) {
val found = msg.keys.sorted().joinToString(", ")
throw IllegalArgumentException(
"A2UI JSONL line $lineNumber: expected exactly one of ${allowed.sorted().joinToString(", ")}; found: $found",
)
}
}
companion object {
const val a2uiReadyCheckJS: String =
"""
(() => {
try {
const host = globalThis.openclawA2UI;
return !!host && typeof host.applyMessages === 'function';
} catch (_) {
return false;
}
})()
"""
const val a2uiResetJS: String =
"""
(() => {
try {
const host = globalThis.openclawA2UI;
if (!host) return { ok: false, error: "missing openclawA2UI" };
return host.reset();
} catch (e) {
return { ok: false, error: String(e?.message ?? e) };
}
})()
"""
fun a2uiApplyMessagesJS(messagesJson: String): String {
return """
(() => {
try {
const host = globalThis.openclawA2UI;
if (!host) return { ok: false, error: "missing openclawA2UI" };
const messages = $messagesJson;
return host.applyMessages(messages);
} catch (e) {
return { ok: false, error: String(e?.message ?? e) };
}
})()
""".trimIndent()
}
}
}

View file

@ -0,0 +1,384 @@
package ai.openclaw.app.node
import android.Manifest
import android.content.ContentResolver
import android.content.ContentUris
import android.content.ContentValues
import android.content.Context
import android.provider.CalendarContract
import androidx.core.content.ContextCompat
import ai.openclaw.app.gateway.GatewaySession
import java.time.Instant
import java.time.temporal.ChronoUnit
import java.util.TimeZone
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.buildJsonArray
import kotlinx.serialization.json.buildJsonObject
import kotlinx.serialization.json.put
private const val DEFAULT_CALENDAR_LIMIT = 50
internal data class CalendarEventsRequest(
val startMs: Long,
val endMs: Long,
val limit: Int,
)
internal data class CalendarAddRequest(
val title: String,
val startMs: Long,
val endMs: Long,
val isAllDay: Boolean,
val location: String?,
val notes: String?,
val calendarId: Long?,
val calendarTitle: String?,
)
internal data class CalendarEventRecord(
val identifier: String,
val title: String,
val startISO: String,
val endISO: String,
val isAllDay: Boolean,
val location: String?,
val calendarTitle: String?,
)
internal interface CalendarDataSource {
fun hasReadPermission(context: Context): Boolean
fun hasWritePermission(context: Context): Boolean
fun events(context: Context, request: CalendarEventsRequest): List<CalendarEventRecord>
fun add(context: Context, request: CalendarAddRequest): CalendarEventRecord
}
private object SystemCalendarDataSource : CalendarDataSource {
override fun hasReadPermission(context: Context): Boolean {
return ContextCompat.checkSelfPermission(context, Manifest.permission.READ_CALENDAR) ==
android.content.pm.PackageManager.PERMISSION_GRANTED
}
override fun hasWritePermission(context: Context): Boolean {
return ContextCompat.checkSelfPermission(context, Manifest.permission.WRITE_CALENDAR) ==
android.content.pm.PackageManager.PERMISSION_GRANTED
}
override fun events(context: Context, request: CalendarEventsRequest): List<CalendarEventRecord> {
val resolver = context.contentResolver
val builder = CalendarContract.Instances.CONTENT_URI.buildUpon()
ContentUris.appendId(builder, request.startMs)
ContentUris.appendId(builder, request.endMs)
val projection =
arrayOf(
CalendarContract.Instances.EVENT_ID,
CalendarContract.Instances.TITLE,
CalendarContract.Instances.BEGIN,
CalendarContract.Instances.END,
CalendarContract.Instances.ALL_DAY,
CalendarContract.Instances.EVENT_LOCATION,
CalendarContract.Instances.CALENDAR_DISPLAY_NAME,
)
val sortOrder = "${CalendarContract.Instances.BEGIN} ASC LIMIT ${request.limit}"
resolver.query(builder.build(), projection, null, null, sortOrder).use { cursor ->
if (cursor == null) return emptyList()
val out = mutableListOf<CalendarEventRecord>()
while (cursor.moveToNext() && out.size < request.limit) {
val id = cursor.getLong(0)
val title = cursor.getString(1)?.trim().orEmpty().ifEmpty { "(untitled)" }
val beginMs = cursor.getLong(2)
val endMs = cursor.getLong(3)
val isAllDay = cursor.getInt(4) == 1
val location = cursor.getString(5)?.trim()?.ifEmpty { null }
val calendarTitle = cursor.getString(6)?.trim()?.ifEmpty { null }
out +=
CalendarEventRecord(
identifier = id.toString(),
title = title,
startISO = Instant.ofEpochMilli(beginMs).toString(),
endISO = Instant.ofEpochMilli(endMs).toString(),
isAllDay = isAllDay,
location = location,
calendarTitle = calendarTitle,
)
}
return out
}
}
override fun add(context: Context, request: CalendarAddRequest): CalendarEventRecord {
val resolver = context.contentResolver
val resolvedCalendarId = resolveCalendarId(resolver, request.calendarId, request.calendarTitle)
val values =
ContentValues().apply {
put(CalendarContract.Events.CALENDAR_ID, resolvedCalendarId)
put(CalendarContract.Events.TITLE, request.title)
put(CalendarContract.Events.DTSTART, request.startMs)
put(CalendarContract.Events.DTEND, request.endMs)
put(CalendarContract.Events.ALL_DAY, if (request.isAllDay) 1 else 0)
put(CalendarContract.Events.EVENT_TIMEZONE, TimeZone.getDefault().id)
request.location?.let { put(CalendarContract.Events.EVENT_LOCATION, it) }
request.notes?.let { put(CalendarContract.Events.DESCRIPTION, it) }
}
val uri = resolver.insert(CalendarContract.Events.CONTENT_URI, values)
?: throw IllegalStateException("calendar insert failed")
val eventId = uri.lastPathSegment?.toLongOrNull()
?: throw IllegalStateException("calendar insert failed")
return loadEventById(resolver, eventId)
?: throw IllegalStateException("calendar insert failed")
}
private fun resolveCalendarId(
resolver: ContentResolver,
calendarId: Long?,
calendarTitle: String?,
): Long {
if (calendarId != null) {
if (calendarExists(resolver, calendarId)) return calendarId
throw IllegalArgumentException("CALENDAR_NOT_FOUND: no calendar id $calendarId")
}
if (!calendarTitle.isNullOrEmpty()) {
findCalendarByTitle(resolver, calendarTitle)?.let { return it }
throw IllegalArgumentException("CALENDAR_NOT_FOUND: no calendar named $calendarTitle")
}
findDefaultCalendarId(resolver)?.let { return it }
throw IllegalArgumentException("CALENDAR_NOT_FOUND: no default calendar")
}
private fun calendarExists(resolver: ContentResolver, id: Long): Boolean {
val projection = arrayOf(CalendarContract.Calendars._ID)
resolver.query(
CalendarContract.Calendars.CONTENT_URI,
projection,
"${CalendarContract.Calendars._ID}=?",
arrayOf(id.toString()),
null,
).use { cursor ->
return cursor != null && cursor.moveToFirst()
}
}
private fun findCalendarByTitle(resolver: ContentResolver, title: String): Long? {
val projection = arrayOf(CalendarContract.Calendars._ID)
resolver.query(
CalendarContract.Calendars.CONTENT_URI,
projection,
"${CalendarContract.Calendars.CALENDAR_DISPLAY_NAME}=?",
arrayOf(title),
"${CalendarContract.Calendars.IS_PRIMARY} DESC",
).use { cursor ->
if (cursor == null || !cursor.moveToFirst()) return null
return cursor.getLong(0)
}
}
private fun findDefaultCalendarId(resolver: ContentResolver): Long? {
val projection = arrayOf(CalendarContract.Calendars._ID)
resolver.query(
CalendarContract.Calendars.CONTENT_URI,
projection,
"${CalendarContract.Calendars.VISIBLE}=1",
null,
"${CalendarContract.Calendars.IS_PRIMARY} DESC, ${CalendarContract.Calendars._ID} ASC",
).use { cursor ->
if (cursor == null || !cursor.moveToFirst()) return null
return cursor.getLong(0)
}
}
private fun loadEventById(
resolver: ContentResolver,
eventId: Long,
): CalendarEventRecord? {
val projection =
arrayOf(
CalendarContract.Events._ID,
CalendarContract.Events.TITLE,
CalendarContract.Events.DTSTART,
CalendarContract.Events.DTEND,
CalendarContract.Events.ALL_DAY,
CalendarContract.Events.EVENT_LOCATION,
CalendarContract.Events.CALENDAR_DISPLAY_NAME,
)
resolver.query(
CalendarContract.Events.CONTENT_URI,
projection,
"${CalendarContract.Events._ID}=?",
arrayOf(eventId.toString()),
null,
).use { cursor ->
if (cursor == null || !cursor.moveToFirst()) return null
return CalendarEventRecord(
identifier = cursor.getLong(0).toString(),
title = cursor.getString(1)?.trim().orEmpty().ifEmpty { "(untitled)" },
startISO = Instant.ofEpochMilli(cursor.getLong(2)).toString(),
endISO = Instant.ofEpochMilli(cursor.getLong(3)).toString(),
isAllDay = cursor.getInt(4) == 1,
location = cursor.getString(5)?.trim()?.ifEmpty { null },
calendarTitle = cursor.getString(6)?.trim()?.ifEmpty { null },
)
}
}
}
class CalendarHandler private constructor(
private val appContext: Context,
private val dataSource: CalendarDataSource,
) {
constructor(appContext: Context) : this(appContext = appContext, dataSource = SystemCalendarDataSource)
fun handleCalendarEvents(paramsJson: String?): GatewaySession.InvokeResult {
if (!dataSource.hasReadPermission(appContext)) {
return GatewaySession.InvokeResult.error(
code = "CALENDAR_PERMISSION_REQUIRED",
message = "CALENDAR_PERMISSION_REQUIRED: grant Calendar permission",
)
}
val request =
parseEventsRequest(paramsJson)
?: return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: expected JSON object",
)
return try {
val events = dataSource.events(appContext, request)
GatewaySession.InvokeResult.ok(
buildJsonObject {
put(
"events",
buildJsonArray { events.forEach { add(eventJson(it)) } },
)
}.toString(),
)
} catch (err: Throwable) {
GatewaySession.InvokeResult.error(
code = "CALENDAR_UNAVAILABLE",
message = "CALENDAR_UNAVAILABLE: ${err.message ?: "calendar query failed"}",
)
}
}
fun handleCalendarAdd(paramsJson: String?): GatewaySession.InvokeResult {
if (!dataSource.hasWritePermission(appContext)) {
return GatewaySession.InvokeResult.error(
code = "CALENDAR_PERMISSION_REQUIRED",
message = "CALENDAR_PERMISSION_REQUIRED: grant Calendar permission",
)
}
val request =
parseAddRequest(paramsJson)
?: return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: expected JSON object",
)
if (request.title.isEmpty()) {
return GatewaySession.InvokeResult.error(
code = "CALENDAR_INVALID",
message = "CALENDAR_INVALID: title required",
)
}
if (request.endMs <= request.startMs) {
return GatewaySession.InvokeResult.error(
code = "CALENDAR_INVALID",
message = "CALENDAR_INVALID: endISO must be after startISO",
)
}
return try {
val event = dataSource.add(appContext, request)
GatewaySession.InvokeResult.ok(
buildJsonObject {
put("event", eventJson(event))
}.toString(),
)
} catch (err: IllegalArgumentException) {
val msg = err.message ?: "CALENDAR_INVALID: invalid request"
val code = if (msg.startsWith("CALENDAR_NOT_FOUND")) "CALENDAR_NOT_FOUND" else "CALENDAR_INVALID"
GatewaySession.InvokeResult.error(code = code, message = msg)
} catch (err: Throwable) {
GatewaySession.InvokeResult.error(
code = "CALENDAR_UNAVAILABLE",
message = "CALENDAR_UNAVAILABLE: ${err.message ?: "calendar add failed"}",
)
}
}
private fun parseEventsRequest(paramsJson: String?): CalendarEventsRequest? {
if (paramsJson.isNullOrBlank()) {
val start = Instant.now()
val end = start.plus(7, ChronoUnit.DAYS)
return CalendarEventsRequest(startMs = start.toEpochMilli(), endMs = end.toEpochMilli(), limit = DEFAULT_CALENDAR_LIMIT)
}
val params =
try {
Json.parseToJsonElement(paramsJson).asObjectOrNull()
} catch (_: Throwable) {
null
} ?: return null
val start = parseISO((params["startISO"] as? JsonPrimitive)?.content)
val end = parseISO((params["endISO"] as? JsonPrimitive)?.content)
val resolvedStart = start ?: Instant.now()
val resolvedEnd = end ?: resolvedStart.plus(7, ChronoUnit.DAYS)
val limit = ((params["limit"] as? JsonPrimitive)?.content?.toIntOrNull() ?: DEFAULT_CALENDAR_LIMIT).coerceIn(1, 500)
return CalendarEventsRequest(
startMs = resolvedStart.toEpochMilli(),
endMs = resolvedEnd.toEpochMilli(),
limit = limit,
)
}
private fun parseAddRequest(paramsJson: String?): CalendarAddRequest? {
val params =
try {
paramsJson?.let { Json.parseToJsonElement(it).asObjectOrNull() }
} catch (_: Throwable) {
null
} ?: return null
val start = parseISO((params["startISO"] as? JsonPrimitive)?.content)
?: return null
val end = parseISO((params["endISO"] as? JsonPrimitive)?.content)
?: return null
return CalendarAddRequest(
title = (params["title"] as? JsonPrimitive)?.content?.trim().orEmpty(),
startMs = start.toEpochMilli(),
endMs = end.toEpochMilli(),
isAllDay = (params["isAllDay"] as? JsonPrimitive)?.content?.toBooleanStrictOrNull() ?: false,
location = (params["location"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
notes = (params["notes"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
calendarId = (params["calendarId"] as? JsonPrimitive)?.content?.toLongOrNull(),
calendarTitle = (params["calendarTitle"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
)
}
private fun parseISO(raw: String?): Instant? {
val value = raw?.trim().orEmpty()
if (value.isEmpty()) return null
return try {
Instant.parse(value)
} catch (_: Throwable) {
null
}
}
private fun eventJson(event: CalendarEventRecord): JsonObject {
return buildJsonObject {
put("identifier", JsonPrimitive(event.identifier))
put("title", JsonPrimitive(event.title))
put("startISO", JsonPrimitive(event.startISO))
put("endISO", JsonPrimitive(event.endISO))
put("isAllDay", JsonPrimitive(event.isAllDay))
event.location?.let { put("location", JsonPrimitive(it)) }
event.calendarTitle?.let { put("calendarTitle", JsonPrimitive(it)) }
}
}
companion object {
internal fun forTesting(
appContext: Context,
dataSource: CalendarDataSource,
): CalendarHandler = CalendarHandler(appContext = appContext, dataSource = dataSource)
}
}

View file

@ -0,0 +1,247 @@
package ai.openclaw.app.node
import android.Manifest
import android.content.Context
import android.provider.CallLog
import androidx.core.content.ContextCompat
import ai.openclaw.app.gateway.GatewaySession
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonArray
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.buildJsonObject
import kotlinx.serialization.json.buildJsonArray
import kotlinx.serialization.json.put
private const val DEFAULT_CALL_LOG_LIMIT = 25
internal data class CallLogRecord(
val number: String?,
val cachedName: String?,
val date: Long,
val duration: Long,
val type: Int,
)
internal data class CallLogSearchRequest(
val limit: Int, // Number of records to return
val offset: Int, // Offset value
val cachedName: String?, // Search by contact name
val number: String?, // Search by phone number
val date: Long?, // Search by time (timestamp, deprecated, use dateStart/dateEnd)
val dateStart: Long?, // Query start time (timestamp)
val dateEnd: Long?, // Query end time (timestamp)
val duration: Long?, // Search by duration (seconds)
val type: Int?, // Search by call log type
)
internal interface CallLogDataSource {
fun hasReadPermission(context: Context): Boolean
fun search(context: Context, request: CallLogSearchRequest): List<CallLogRecord>
}
private object SystemCallLogDataSource : CallLogDataSource {
override fun hasReadPermission(context: Context): Boolean {
return ContextCompat.checkSelfPermission(
context,
Manifest.permission.READ_CALL_LOG
) == android.content.pm.PackageManager.PERMISSION_GRANTED
}
override fun search(context: Context, request: CallLogSearchRequest): List<CallLogRecord> {
val resolver = context.contentResolver
val projection = arrayOf(
CallLog.Calls.NUMBER,
CallLog.Calls.CACHED_NAME,
CallLog.Calls.DATE,
CallLog.Calls.DURATION,
CallLog.Calls.TYPE,
)
// Build selection and selectionArgs for filtering
val selections = mutableListOf<String>()
val selectionArgs = mutableListOf<String>()
request.cachedName?.let {
selections.add("${CallLog.Calls.CACHED_NAME} LIKE ?")
selectionArgs.add("%$it%")
}
request.number?.let {
selections.add("${CallLog.Calls.NUMBER} LIKE ?")
selectionArgs.add("%$it%")
}
// Support time range query
if (request.dateStart != null && request.dateEnd != null) {
selections.add("${CallLog.Calls.DATE} >= ? AND ${CallLog.Calls.DATE} <= ?")
selectionArgs.add(request.dateStart.toString())
selectionArgs.add(request.dateEnd.toString())
} else if (request.dateStart != null) {
selections.add("${CallLog.Calls.DATE} >= ?")
selectionArgs.add(request.dateStart.toString())
} else if (request.dateEnd != null) {
selections.add("${CallLog.Calls.DATE} <= ?")
selectionArgs.add(request.dateEnd.toString())
} else if (request.date != null) {
// Compatible with the old date parameter (exact match)
selections.add("${CallLog.Calls.DATE} = ?")
selectionArgs.add(request.date.toString())
}
request.duration?.let {
selections.add("${CallLog.Calls.DURATION} = ?")
selectionArgs.add(it.toString())
}
request.type?.let {
selections.add("${CallLog.Calls.TYPE} = ?")
selectionArgs.add(it.toString())
}
val selection = if (selections.isNotEmpty()) selections.joinToString(" AND ") else null
val selectionArgsArray = if (selectionArgs.isNotEmpty()) selectionArgs.toTypedArray() else null
val sortOrder = "${CallLog.Calls.DATE} DESC"
resolver.query(
CallLog.Calls.CONTENT_URI,
projection,
selection,
selectionArgsArray,
sortOrder,
).use { cursor ->
if (cursor == null) return emptyList()
val numberIndex = cursor.getColumnIndex(CallLog.Calls.NUMBER)
val cachedNameIndex = cursor.getColumnIndex(CallLog.Calls.CACHED_NAME)
val dateIndex = cursor.getColumnIndex(CallLog.Calls.DATE)
val durationIndex = cursor.getColumnIndex(CallLog.Calls.DURATION)
val typeIndex = cursor.getColumnIndex(CallLog.Calls.TYPE)
// Skip offset rows
if (request.offset > 0 && cursor.moveToPosition(request.offset - 1)) {
// Successfully moved to offset position
}
val out = mutableListOf<CallLogRecord>()
var count = 0
while (cursor.moveToNext() && count < request.limit) {
out += CallLogRecord(
number = cursor.getString(numberIndex),
cachedName = cursor.getString(cachedNameIndex),
date = cursor.getLong(dateIndex),
duration = cursor.getLong(durationIndex),
type = cursor.getInt(typeIndex),
)
count++
}
return out
}
}
}
class CallLogHandler private constructor(
private val appContext: Context,
private val dataSource: CallLogDataSource,
) {
constructor(appContext: Context) : this(appContext = appContext, dataSource = SystemCallLogDataSource)
fun handleCallLogSearch(paramsJson: String?): GatewaySession.InvokeResult {
if (!dataSource.hasReadPermission(appContext)) {
return GatewaySession.InvokeResult.error(
code = "CALL_LOG_PERMISSION_REQUIRED",
message = "CALL_LOG_PERMISSION_REQUIRED: grant Call Log permission",
)
}
val request = parseSearchRequest(paramsJson)
?: return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: expected JSON object",
)
return try {
val callLogs = dataSource.search(appContext, request)
GatewaySession.InvokeResult.ok(
buildJsonObject {
put(
"callLogs",
buildJsonArray {
callLogs.forEach { add(callLogJson(it)) }
},
)
}.toString(),
)
} catch (err: Throwable) {
GatewaySession.InvokeResult.error(
code = "CALL_LOG_UNAVAILABLE",
message = "CALL_LOG_UNAVAILABLE: ${err.message ?: "call log query failed"}",
)
}
}
private fun parseSearchRequest(paramsJson: String?): CallLogSearchRequest? {
if (paramsJson.isNullOrBlank()) {
return CallLogSearchRequest(
limit = DEFAULT_CALL_LOG_LIMIT,
offset = 0,
cachedName = null,
number = null,
date = null,
dateStart = null,
dateEnd = null,
duration = null,
type = null,
)
}
val params = try {
Json.parseToJsonElement(paramsJson).asObjectOrNull()
} catch (_: Throwable) {
null
} ?: return null
val limit = ((params["limit"] as? JsonPrimitive)?.content?.toIntOrNull() ?: DEFAULT_CALL_LOG_LIMIT)
.coerceIn(1, 200)
val offset = ((params["offset"] as? JsonPrimitive)?.content?.toIntOrNull() ?: 0)
.coerceAtLeast(0)
val cachedName = (params["cachedName"] as? JsonPrimitive)?.content?.takeIf { it.isNotBlank() }
val number = (params["number"] as? JsonPrimitive)?.content?.takeIf { it.isNotBlank() }
val date = (params["date"] as? JsonPrimitive)?.content?.toLongOrNull()
val dateStart = (params["dateStart"] as? JsonPrimitive)?.content?.toLongOrNull()
val dateEnd = (params["dateEnd"] as? JsonPrimitive)?.content?.toLongOrNull()
val duration = (params["duration"] as? JsonPrimitive)?.content?.toLongOrNull()
val type = (params["type"] as? JsonPrimitive)?.content?.toIntOrNull()
return CallLogSearchRequest(
limit = limit,
offset = offset,
cachedName = cachedName,
number = number,
date = date,
dateStart = dateStart,
dateEnd = dateEnd,
duration = duration,
type = type,
)
}
private fun callLogJson(callLog: CallLogRecord): JsonObject {
return buildJsonObject {
put("number", JsonPrimitive(callLog.number))
put("cachedName", JsonPrimitive(callLog.cachedName))
put("date", JsonPrimitive(callLog.date))
put("duration", JsonPrimitive(callLog.duration))
put("type", JsonPrimitive(callLog.type))
}
}
companion object {
internal fun forTesting(
appContext: Context,
dataSource: CallLogDataSource,
): CallLogHandler = CallLogHandler(appContext = appContext, dataSource = dataSource)
}
}

View file

@ -0,0 +1,425 @@
package ai.openclaw.app.node
import android.Manifest
import android.annotation.SuppressLint
import android.content.Context
import android.graphics.Bitmap
import android.graphics.BitmapFactory
import android.graphics.Matrix
import android.content.pm.PackageManager
import android.hardware.camera2.CameraCharacteristics
import android.util.Base64
import androidx.camera.camera2.interop.Camera2CameraInfo
import androidx.camera.core.CameraInfo
import androidx.exifinterface.media.ExifInterface
import androidx.lifecycle.LifecycleOwner
import androidx.camera.core.CameraSelector
import androidx.camera.core.ImageCapture
import androidx.camera.core.ImageCaptureException
import androidx.camera.lifecycle.ProcessCameraProvider
import androidx.camera.video.FileOutputOptions
import androidx.camera.video.FallbackStrategy
import androidx.camera.video.Quality
import androidx.camera.video.QualitySelector
import androidx.camera.video.Recorder
import androidx.camera.video.Recording
import androidx.camera.video.VideoCapture
import androidx.camera.video.VideoRecordEvent
import androidx.core.content.ContextCompat
import androidx.core.content.ContextCompat.checkSelfPermission
import androidx.core.graphics.scale
import ai.openclaw.app.PermissionRequester
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.suspendCancellableCoroutine
import kotlinx.coroutines.withTimeout
import kotlinx.coroutines.withContext
import kotlinx.serialization.json.JsonObject
import java.io.ByteArrayOutputStream
import java.io.File
import java.util.concurrent.Executor
import kotlin.math.roundToInt
import kotlin.coroutines.resume
import kotlin.coroutines.resumeWithException
class CameraCaptureManager(private val context: Context) {
data class Payload(val payloadJson: String)
data class FilePayload(val file: File, val durationMs: Long, val hasAudio: Boolean)
data class CameraDeviceInfo(
val id: String,
val name: String,
val position: String,
val deviceType: String,
)
@Volatile private var lifecycleOwner: LifecycleOwner? = null
@Volatile private var permissionRequester: PermissionRequester? = null
fun attachLifecycleOwner(owner: LifecycleOwner) {
lifecycleOwner = owner
}
fun attachPermissionRequester(requester: PermissionRequester) {
permissionRequester = requester
}
suspend fun listDevices(): List<CameraDeviceInfo> =
withContext(Dispatchers.Main) {
val provider = context.cameraProvider()
provider.availableCameraInfos
.mapNotNull { info -> cameraDeviceInfoOrNull(info) }
.sortedBy { it.id }
}
private suspend fun ensureCameraPermission() {
val granted = checkSelfPermission(context, Manifest.permission.CAMERA) == PackageManager.PERMISSION_GRANTED
if (granted) return
val requester = permissionRequester
?: throw IllegalStateException("CAMERA_PERMISSION_REQUIRED: grant Camera permission")
val results = requester.requestIfMissing(listOf(Manifest.permission.CAMERA))
if (results[Manifest.permission.CAMERA] != true) {
throw IllegalStateException("CAMERA_PERMISSION_REQUIRED: grant Camera permission")
}
}
private suspend fun ensureMicPermission() {
val granted = checkSelfPermission(context, Manifest.permission.RECORD_AUDIO) == PackageManager.PERMISSION_GRANTED
if (granted) return
val requester = permissionRequester
?: throw IllegalStateException("MIC_PERMISSION_REQUIRED: grant Microphone permission")
val results = requester.requestIfMissing(listOf(Manifest.permission.RECORD_AUDIO))
if (results[Manifest.permission.RECORD_AUDIO] != true) {
throw IllegalStateException("MIC_PERMISSION_REQUIRED: grant Microphone permission")
}
}
suspend fun snap(paramsJson: String?): Payload =
withContext(Dispatchers.Main) {
ensureCameraPermission()
val owner = lifecycleOwner ?: throw IllegalStateException("UNAVAILABLE: camera not ready")
val params = parseJsonParamsObject(paramsJson)
val facing = parseFacing(params) ?: "front"
val quality = (parseQuality(params) ?: 0.95).coerceIn(0.1, 1.0)
val maxWidth = parseMaxWidth(params) ?: 1600
val deviceId = parseDeviceId(params)
val provider = context.cameraProvider()
val capture = ImageCapture.Builder().build()
val selector = resolveCameraSelector(provider, facing, deviceId)
provider.unbindAll()
provider.bindToLifecycle(owner, selector, capture)
val (bytes, orientation) = capture.takeJpegWithExif(context.mainExecutor())
val decoded = BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
?: throw IllegalStateException("UNAVAILABLE: failed to decode captured image")
val rotated = rotateBitmapByExif(decoded, orientation)
val scaled =
if (maxWidth > 0 && rotated.width > maxWidth) {
val h =
(rotated.height.toDouble() * (maxWidth.toDouble() / rotated.width.toDouble()))
.toInt()
.coerceAtLeast(1)
val s = rotated.scale(maxWidth, h)
if (s !== rotated) rotated.recycle()
s
} else {
rotated
}
try {
val maxPayloadBytes = 5 * 1024 * 1024
// Base64 inflates payloads by ~4/3; cap encoded bytes so the payload stays under 5MB (API limit).
val maxEncodedBytes = (maxPayloadBytes / 4) * 3
val result =
JpegSizeLimiter.compressToLimit(
initialWidth = scaled.width,
initialHeight = scaled.height,
startQuality = (quality * 100.0).roundToInt().coerceIn(10, 100),
maxBytes = maxEncodedBytes,
encode = { width, height, q ->
val bitmap =
if (width == scaled.width && height == scaled.height) {
scaled
} else {
scaled.scale(width, height)
}
val out = ByteArrayOutputStream()
if (!bitmap.compress(Bitmap.CompressFormat.JPEG, q, out)) {
if (bitmap !== scaled) bitmap.recycle()
throw IllegalStateException("UNAVAILABLE: failed to encode JPEG")
}
if (bitmap !== scaled) {
bitmap.recycle()
}
out.toByteArray()
},
)
val base64 = Base64.encodeToString(result.bytes, Base64.NO_WRAP)
Payload(
"""{"format":"jpg","base64":"$base64","width":${result.width},"height":${result.height}}""",
)
} finally {
scaled.recycle()
}
}
@SuppressLint("MissingPermission")
suspend fun clip(paramsJson: String?): FilePayload =
withContext(Dispatchers.Main) {
ensureCameraPermission()
val owner = lifecycleOwner ?: throw IllegalStateException("UNAVAILABLE: camera not ready")
val params = parseJsonParamsObject(paramsJson)
val facing = parseFacing(params) ?: "front"
val durationMs = (parseDurationMs(params) ?: 3_000).coerceIn(200, 60_000)
val includeAudio = parseIncludeAudio(params) ?: true
val deviceId = parseDeviceId(params)
if (includeAudio) ensureMicPermission()
android.util.Log.w("CameraCaptureManager", "clip: start facing=$facing duration=$durationMs audio=$includeAudio deviceId=${deviceId ?: "-"}")
val provider = context.cameraProvider()
android.util.Log.w("CameraCaptureManager", "clip: got camera provider")
// Use LOWEST quality for smallest files over WebSocket
val recorder = Recorder.Builder()
.setQualitySelector(
QualitySelector.from(Quality.LOWEST, FallbackStrategy.lowerQualityOrHigherThan(Quality.LOWEST))
)
.build()
val videoCapture = VideoCapture.withOutput(recorder)
val selector = resolveCameraSelector(provider, facing, deviceId)
// CameraX requires a Preview use case for the camera to start producing frames;
// without it, the encoder may get no data (ERROR_NO_VALID_DATA).
val preview = androidx.camera.core.Preview.Builder().build()
// Provide a dummy SurfaceTexture so the preview pipeline activates
val surfaceTexture = android.graphics.SurfaceTexture(0)
surfaceTexture.setDefaultBufferSize(640, 480)
preview.setSurfaceProvider { request ->
val surface = android.view.Surface(surfaceTexture)
request.provideSurface(surface, context.mainExecutor()) { result ->
surface.release()
surfaceTexture.release()
}
}
provider.unbindAll()
android.util.Log.w("CameraCaptureManager", "clip: binding preview + videoCapture to lifecycle")
val camera = provider.bindToLifecycle(owner, selector, preview, videoCapture)
android.util.Log.w("CameraCaptureManager", "clip: bound, cameraInfo=${camera.cameraInfo}")
// Give camera pipeline time to initialize before recording
android.util.Log.w("CameraCaptureManager", "clip: warming up camera 1.5s...")
kotlinx.coroutines.delay(1_500)
val file = File.createTempFile("openclaw-clip-", ".mp4")
val outputOptions = FileOutputOptions.Builder(file).build()
val finalized = kotlinx.coroutines.CompletableDeferred<VideoRecordEvent.Finalize>()
android.util.Log.w("CameraCaptureManager", "clip: starting recording to ${file.absolutePath}")
val recording: Recording =
videoCapture.output
.prepareRecording(context, outputOptions)
.apply {
if (includeAudio) withAudioEnabled()
}
.start(context.mainExecutor()) { event ->
android.util.Log.w("CameraCaptureManager", "clip: event ${event.javaClass.simpleName}")
if (event is VideoRecordEvent.Status) {
android.util.Log.w("CameraCaptureManager", "clip: recording status update")
}
if (event is VideoRecordEvent.Finalize) {
android.util.Log.w("CameraCaptureManager", "clip: finalize hasError=${event.hasError()} error=${event.error} cause=${event.cause}")
finalized.complete(event)
}
}
android.util.Log.w("CameraCaptureManager", "clip: recording started, delaying ${durationMs}ms")
try {
kotlinx.coroutines.delay(durationMs.toLong())
} finally {
android.util.Log.w("CameraCaptureManager", "clip: stopping recording")
recording.stop()
}
val finalizeEvent =
try {
withTimeout(15_000) { finalized.await() }
} catch (err: Throwable) {
android.util.Log.e("CameraCaptureManager", "clip: finalize timed out", err)
withContext(Dispatchers.IO) { file.delete() }
provider.unbindAll()
throw IllegalStateException("UNAVAILABLE: camera clip finalize timed out")
}
if (finalizeEvent.hasError()) {
android.util.Log.e("CameraCaptureManager", "clip: FAILED error=${finalizeEvent.error}, cause=${finalizeEvent.cause}", finalizeEvent.cause)
// Check file size for debugging
val fileSize = withContext(Dispatchers.IO) { if (file.exists()) file.length() else -1 }
android.util.Log.e("CameraCaptureManager", "clip: file exists=${file.exists()} size=$fileSize")
withContext(Dispatchers.IO) { file.delete() }
provider.unbindAll()
throw IllegalStateException("UNAVAILABLE: camera clip failed (error=${finalizeEvent.error})")
}
val fileSize = withContext(Dispatchers.IO) { file.length() }
android.util.Log.w("CameraCaptureManager", "clip: SUCCESS file size=$fileSize")
provider.unbindAll()
FilePayload(file = file, durationMs = durationMs.toLong(), hasAudio = includeAudio)
}
private fun rotateBitmapByExif(bitmap: Bitmap, orientation: Int): Bitmap {
val matrix = Matrix()
when (orientation) {
ExifInterface.ORIENTATION_ROTATE_90 -> matrix.postRotate(90f)
ExifInterface.ORIENTATION_ROTATE_180 -> matrix.postRotate(180f)
ExifInterface.ORIENTATION_ROTATE_270 -> matrix.postRotate(270f)
ExifInterface.ORIENTATION_FLIP_HORIZONTAL -> matrix.postScale(-1f, 1f)
ExifInterface.ORIENTATION_FLIP_VERTICAL -> matrix.postScale(1f, -1f)
ExifInterface.ORIENTATION_TRANSPOSE -> {
matrix.postRotate(90f)
matrix.postScale(-1f, 1f)
}
ExifInterface.ORIENTATION_TRANSVERSE -> {
matrix.postRotate(-90f)
matrix.postScale(-1f, 1f)
}
else -> return bitmap
}
val rotated = Bitmap.createBitmap(bitmap, 0, 0, bitmap.width, bitmap.height, matrix, true)
if (rotated !== bitmap) {
bitmap.recycle()
}
return rotated
}
private fun parseFacing(params: JsonObject?): String? {
val value = parseJsonString(params, "facing")?.trim()?.lowercase() ?: return null
return when (value) {
"front", "back" -> value
else -> null
}
}
private fun parseQuality(params: JsonObject?): Double? =
parseJsonDouble(params, "quality")
private fun parseMaxWidth(params: JsonObject?): Int? =
parseJsonInt(params, "maxWidth")
?.takeIf { it > 0 }
private fun parseDurationMs(params: JsonObject?): Int? =
parseJsonInt(params, "durationMs")
private fun parseDeviceId(params: JsonObject?): String? =
parseJsonString(params, "deviceId")
?.trim()
?.takeIf { it.isNotEmpty() }
private fun parseIncludeAudio(params: JsonObject?): Boolean? = parseJsonBooleanFlag(params, "includeAudio")
private fun Context.mainExecutor(): Executor = ContextCompat.getMainExecutor(this)
private fun resolveCameraSelector(
provider: ProcessCameraProvider,
facing: String,
deviceId: String?,
): CameraSelector {
if (deviceId.isNullOrEmpty()) {
return if (facing == "front") CameraSelector.DEFAULT_FRONT_CAMERA else CameraSelector.DEFAULT_BACK_CAMERA
}
val availableIds = provider.availableCameraInfos.mapNotNull { cameraIdOrNull(it) }.toSet()
if (!availableIds.contains(deviceId)) {
throw IllegalStateException("INVALID_REQUEST: unknown camera deviceId '$deviceId'")
}
return CameraSelector.Builder()
.addCameraFilter { infos -> infos.filter { cameraIdOrNull(it) == deviceId } }
.build()
}
@SuppressLint("UnsafeOptInUsageError")
private fun cameraDeviceInfoOrNull(info: CameraInfo): CameraDeviceInfo? {
val cameraId = cameraIdOrNull(info) ?: return null
val lensFacing =
runCatching {
Camera2CameraInfo.from(info).getCameraCharacteristic(CameraCharacteristics.LENS_FACING)
}.getOrNull()
val position =
when (lensFacing) {
CameraCharacteristics.LENS_FACING_FRONT -> "front"
CameraCharacteristics.LENS_FACING_BACK -> "back"
CameraCharacteristics.LENS_FACING_EXTERNAL -> "external"
else -> "unspecified"
}
val deviceType =
if (lensFacing == CameraCharacteristics.LENS_FACING_EXTERNAL) "external" else "builtIn"
val name =
when (position) {
"front" -> "Front Camera"
"back" -> "Back Camera"
"external" -> "External Camera"
else -> "Camera $cameraId"
}
return CameraDeviceInfo(
id = cameraId,
name = name,
position = position,
deviceType = deviceType,
)
}
@SuppressLint("UnsafeOptInUsageError")
private fun cameraIdOrNull(info: CameraInfo): String? =
runCatching { Camera2CameraInfo.from(info).cameraId }.getOrNull()
}
private suspend fun Context.cameraProvider(): ProcessCameraProvider =
suspendCancellableCoroutine { cont ->
val future = ProcessCameraProvider.getInstance(this)
future.addListener(
{
try {
cont.resume(future.get())
} catch (e: Exception) {
cont.resumeWithException(e)
}
},
ContextCompat.getMainExecutor(this),
)
}
/** Returns (jpegBytes, exifOrientation) so caller can rotate the decoded bitmap. */
private suspend fun ImageCapture.takeJpegWithExif(executor: Executor): Pair<ByteArray, Int> =
suspendCancellableCoroutine { cont ->
val file = File.createTempFile("openclaw-snap-", ".jpg")
val options = ImageCapture.OutputFileOptions.Builder(file).build()
takePicture(
options,
executor,
object : ImageCapture.OnImageSavedCallback {
override fun onError(exception: ImageCaptureException) {
file.delete()
cont.resumeWithException(exception)
}
override fun onImageSaved(outputFileResults: ImageCapture.OutputFileResults) {
try {
val exif = ExifInterface(file.absolutePath)
val orientation = exif.getAttributeInt(
ExifInterface.TAG_ORIENTATION,
ExifInterface.ORIENTATION_NORMAL,
)
val bytes = file.readBytes()
cont.resume(Pair(bytes, orientation))
} catch (e: Exception) {
cont.resumeWithException(e)
} finally {
file.delete()
}
}
},
)
}

View file

@ -0,0 +1,177 @@
package ai.openclaw.app.node
import android.content.Context
import ai.openclaw.app.CameraHudKind
import ai.openclaw.app.BuildConfig
import ai.openclaw.app.gateway.GatewaySession
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.withContext
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.buildJsonArray
import kotlinx.serialization.json.buildJsonObject
import kotlinx.serialization.json.contentOrNull
import kotlinx.serialization.json.put
internal const val CAMERA_CLIP_MAX_RAW_BYTES: Long = 18L * 1024L * 1024L
internal fun isCameraClipWithinPayloadLimit(rawBytes: Long): Boolean =
rawBytes in 0L..CAMERA_CLIP_MAX_RAW_BYTES
class CameraHandler(
private val appContext: Context,
private val camera: CameraCaptureManager,
private val externalAudioCaptureActive: MutableStateFlow<Boolean>,
private val showCameraHud: (message: String, kind: CameraHudKind, autoHideMs: Long?) -> Unit,
private val triggerCameraFlash: () -> Unit,
private val invokeErrorFromThrowable: (err: Throwable) -> Pair<String, String>,
) {
suspend fun handleList(_paramsJson: String?): GatewaySession.InvokeResult {
return try {
val devices = camera.listDevices()
val payload =
buildJsonObject {
put(
"devices",
buildJsonArray {
devices.forEach { device ->
add(
buildJsonObject {
put("id", JsonPrimitive(device.id))
put("name", JsonPrimitive(device.name))
put("position", JsonPrimitive(device.position))
put("deviceType", JsonPrimitive(device.deviceType))
},
)
}
},
)
}.toString()
GatewaySession.InvokeResult.ok(payload)
} catch (err: Throwable) {
val (code, message) = invokeErrorFromThrowable(err)
GatewaySession.InvokeResult.error(code = code, message = message)
}
}
suspend fun handleSnap(paramsJson: String?): GatewaySession.InvokeResult {
val logFile = if (BuildConfig.DEBUG) java.io.File(appContext.cacheDir, "camera_debug.log") else null
fun camLog(msg: String) {
if (!BuildConfig.DEBUG) return
val ts = java.text.SimpleDateFormat("HH:mm:ss.SSS", java.util.Locale.US).format(java.util.Date())
logFile?.appendText("[$ts] $msg\n")
android.util.Log.w("openclaw", "camera.snap: $msg")
}
try {
logFile?.writeText("") // clear
camLog("starting, params=$paramsJson")
camLog("calling showCameraHud")
showCameraHud("Taking photo…", CameraHudKind.Photo, null)
camLog("calling triggerCameraFlash")
triggerCameraFlash()
val res =
try {
camLog("calling camera.snap()")
val r = camera.snap(paramsJson)
camLog("success, payload size=${r.payloadJson.length}")
r
} catch (err: Throwable) {
camLog("inner error: ${err::class.java.simpleName}: ${err.message}")
camLog("stack: ${err.stackTraceToString().take(2000)}")
val (code, message) = invokeErrorFromThrowable(err)
showCameraHud(message, CameraHudKind.Error, 2200)
return GatewaySession.InvokeResult.error(code = code, message = message)
}
camLog("returning result")
showCameraHud("Photo captured", CameraHudKind.Success, 1600)
return GatewaySession.InvokeResult.ok(res.payloadJson)
} catch (err: Throwable) {
camLog("outer error: ${err::class.java.simpleName}: ${err.message}")
camLog("stack: ${err.stackTraceToString().take(2000)}")
return GatewaySession.InvokeResult.error(code = "UNAVAILABLE", message = err.message ?: "camera snap failed")
}
}
suspend fun handleClip(paramsJson: String?): GatewaySession.InvokeResult {
val clipLogFile = if (BuildConfig.DEBUG) java.io.File(appContext.cacheDir, "camera_debug.log") else null
fun clipLog(msg: String) {
if (!BuildConfig.DEBUG) return
val ts = java.text.SimpleDateFormat("HH:mm:ss.SSS", java.util.Locale.US).format(java.util.Date())
clipLogFile?.appendText("[CLIP $ts] $msg\n")
android.util.Log.w("openclaw", "camera.clip: $msg")
}
val includeAudio = parseIncludeAudio(paramsJson) ?: true
if (includeAudio) externalAudioCaptureActive.value = true
try {
clipLogFile?.writeText("") // clear
clipLog("starting, params=$paramsJson includeAudio=$includeAudio")
clipLog("calling showCameraHud")
showCameraHud("Recording…", CameraHudKind.Recording, null)
val filePayload =
try {
clipLog("calling camera.clip()")
val r = camera.clip(paramsJson)
clipLog("success, file size=${r.file.length()}")
r
} catch (err: Throwable) {
clipLog("inner error: ${err::class.java.simpleName}: ${err.message}")
clipLog("stack: ${err.stackTraceToString().take(2000)}")
val (code, message) = invokeErrorFromThrowable(err)
showCameraHud(message, CameraHudKind.Error, 2400)
return GatewaySession.InvokeResult.error(code = code, message = message)
}
val rawBytes = filePayload.file.length()
if (!isCameraClipWithinPayloadLimit(rawBytes)) {
clipLog("payload too large: bytes=$rawBytes max=$CAMERA_CLIP_MAX_RAW_BYTES")
withContext(Dispatchers.IO) { filePayload.file.delete() }
showCameraHud("Clip too large", CameraHudKind.Error, 2400)
return GatewaySession.InvokeResult.error(
code = "PAYLOAD_TOO_LARGE",
message =
"PAYLOAD_TOO_LARGE: camera clip is $rawBytes bytes; max is $CAMERA_CLIP_MAX_RAW_BYTES bytes. Reduce durationMs and retry.",
)
}
val bytes = withContext(Dispatchers.IO) {
try {
filePayload.file.readBytes()
} finally {
filePayload.file.delete()
}
}
val base64 = android.util.Base64.encodeToString(bytes, android.util.Base64.NO_WRAP)
clipLog("returning base64 payload")
showCameraHud("Clip captured", CameraHudKind.Success, 1800)
return GatewaySession.InvokeResult.ok(
"""{"format":"mp4","base64":"$base64","durationMs":${filePayload.durationMs},"hasAudio":${filePayload.hasAudio}}"""
)
} catch (err: Throwable) {
clipLog("outer error: ${err::class.java.simpleName}: ${err.message}")
clipLog("stack: ${err.stackTraceToString().take(2000)}")
return GatewaySession.InvokeResult.error(code = "UNAVAILABLE", message = err.message ?: "camera clip failed")
} finally {
if (includeAudio) externalAudioCaptureActive.value = false
}
}
private fun parseIncludeAudio(paramsJson: String?): Boolean? {
if (paramsJson.isNullOrBlank()) return null
val root =
try {
Json.parseToJsonElement(paramsJson).asObjectOrNull()
} catch (_: Throwable) {
null
} ?: return null
val value =
(root["includeAudio"] as? JsonPrimitive)
?.contentOrNull
?.trim()
?.lowercase()
return when (value) {
"true" -> true
"false" -> false
else -> null
}
}
}

View file

@ -0,0 +1,51 @@
package ai.openclaw.app.node
import java.net.URI
object CanvasActionTrust {
const val scaffoldAssetUrl: String = "file:///android_asset/CanvasScaffold/scaffold.html"
fun isTrustedCanvasActionUrl(rawUrl: String?, trustedA2uiUrls: List<String>): Boolean {
val candidate = rawUrl?.trim().orEmpty()
if (candidate.isEmpty()) return false
if (candidate == scaffoldAssetUrl) return true
val candidateUri = parseUri(candidate) ?: return false
if (candidateUri.scheme.equals("file", ignoreCase = true)) {
return false
}
val normalizedCandidate = normalizeTrustedRemoteA2uiUri(candidateUri) ?: return false
return trustedA2uiUrls.any { trusted ->
matchesTrustedRemoteA2uiUrlExact(normalizedCandidate, trusted)
}
}
private fun matchesTrustedRemoteA2uiUrlExact(candidateUri: URI, trustedUrl: String): Boolean {
val trustedUri = parseUri(trustedUrl) ?: return false
val normalizedTrusted = normalizeTrustedRemoteA2uiUri(trustedUri) ?: return false
return candidateUri == normalizedTrusted
}
private fun normalizeTrustedRemoteA2uiUri(uri: URI): URI? {
// Keep Android trust normalization aligned with iOS ScreenController:
// exact remote URL match, scheme/host normalized, fragment ignored.
val scheme = uri.scheme?.lowercase() ?: return null
if (scheme != "http" && scheme != "https") return null
val host = uri.host?.trim()?.takeIf { it.isNotEmpty() }?.lowercase() ?: return null
return try {
URI(scheme, uri.userInfo, host, uri.port, uri.rawPath, uri.rawQuery, null)
} catch (_: Throwable) {
null
}
}
private fun parseUri(raw: String): URI? =
try {
URI(raw)
} catch (_: Throwable) {
null
}
}

View file

@ -0,0 +1,310 @@
package ai.openclaw.app.node
import android.graphics.Bitmap
import android.graphics.Canvas
import android.os.Looper
import android.util.Log
import android.webkit.WebView
import androidx.core.graphics.createBitmap
import androidx.core.graphics.scale
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.suspendCancellableCoroutine
import kotlinx.coroutines.withContext
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asStateFlow
import java.io.ByteArrayOutputStream
import android.util.Base64
import org.json.JSONObject
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonElement
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import ai.openclaw.app.BuildConfig
import kotlin.coroutines.resume
class CanvasController {
enum class SnapshotFormat(val rawValue: String) {
Png("png"),
Jpeg("jpeg"),
}
@Volatile private var webView: WebView? = null
@Volatile private var url: String? = null
@Volatile private var debugStatusEnabled: Boolean = false
@Volatile private var debugStatusTitle: String? = null
@Volatile private var debugStatusSubtitle: String? = null
@Volatile private var homeCanvasStateJson: String? = null
private val _currentUrl = MutableStateFlow<String?>(null)
val currentUrl: StateFlow<String?> = _currentUrl.asStateFlow()
private val scaffoldAssetUrl = "file:///android_asset/CanvasScaffold/scaffold.html"
private fun clampJpegQuality(quality: Double?): Int {
val q = (quality ?: 0.82).coerceIn(0.1, 1.0)
return (q * 100.0).toInt().coerceIn(1, 100)
}
private fun Bitmap.scaleForMaxWidth(maxWidth: Int?): Bitmap {
if (maxWidth == null || maxWidth <= 0 || width <= maxWidth) {
return this
}
val scaledHeight = (height.toDouble() * (maxWidth.toDouble() / width.toDouble())).toInt().coerceAtLeast(1)
return scale(maxWidth, scaledHeight)
}
fun attach(webView: WebView) {
this.webView = webView
reload()
applyDebugStatus()
applyHomeCanvasState()
}
fun detach(webView: WebView) {
if (this.webView === webView) {
this.webView = null
}
}
fun navigate(url: String) {
val trimmed = url.trim()
this.url = if (trimmed.isBlank() || trimmed == "/") null else trimmed
_currentUrl.value = this.url
reload()
}
fun currentUrl(): String? = url
fun isDefaultCanvas(): Boolean = url == null
fun setDebugStatusEnabled(enabled: Boolean) {
debugStatusEnabled = enabled
applyDebugStatus()
}
fun setDebugStatus(title: String?, subtitle: String?) {
debugStatusTitle = title
debugStatusSubtitle = subtitle
applyDebugStatus()
}
fun onPageFinished() {
applyDebugStatus()
applyHomeCanvasState()
}
fun updateHomeCanvasState(json: String?) {
homeCanvasStateJson = json
applyHomeCanvasState()
}
private inline fun withWebViewOnMain(crossinline block: (WebView) -> Unit) {
val wv = webView ?: return
if (Looper.myLooper() == Looper.getMainLooper()) {
block(wv)
} else {
wv.post { block(wv) }
}
}
private fun reload() {
val currentUrl = url
withWebViewOnMain { wv ->
if (currentUrl == null) {
if (BuildConfig.DEBUG) {
Log.d("OpenClawCanvas", "load scaffold: $scaffoldAssetUrl")
}
wv.loadUrl(scaffoldAssetUrl)
} else {
if (BuildConfig.DEBUG) {
Log.d("OpenClawCanvas", "load url: $currentUrl")
}
wv.loadUrl(currentUrl)
}
}
}
private fun applyDebugStatus() {
val enabled = debugStatusEnabled
val title = debugStatusTitle
val subtitle = debugStatusSubtitle
withWebViewOnMain { wv ->
val titleJs = title?.let { JSONObject.quote(it) } ?: "null"
val subtitleJs = subtitle?.let { JSONObject.quote(it) } ?: "null"
val js = """
(() => {
try {
const api = globalThis.__openclaw;
if (!api) return;
if (typeof api.setDebugStatusEnabled === 'function') {
api.setDebugStatusEnabled(${if (enabled) "true" else "false"});
}
if (!${if (enabled) "true" else "false"}) return;
if (typeof api.setStatus === 'function') {
api.setStatus($titleJs, $subtitleJs);
}
} catch (_) {}
})();
""".trimIndent()
wv.evaluateJavascript(js, null)
}
}
private fun applyHomeCanvasState() {
val payload = homeCanvasStateJson ?: "null"
withWebViewOnMain { wv ->
val js = """
(() => {
try {
const api = globalThis.__openclaw;
if (!api || typeof api.renderHome !== 'function') return;
api.renderHome($payload);
} catch (_) {}
})();
""".trimIndent()
wv.evaluateJavascript(js, null)
}
}
suspend fun eval(javaScript: String): String =
withContext(Dispatchers.Main) {
val wv = webView ?: throw IllegalStateException("no webview")
suspendCancellableCoroutine { cont ->
wv.evaluateJavascript(javaScript) { result ->
cont.resume(result ?: "")
}
}
}
suspend fun snapshotPngBase64(maxWidth: Int?): String =
withContext(Dispatchers.Main) {
val wv = webView ?: throw IllegalStateException("no webview")
val bmp = wv.captureBitmap()
try {
val scaled = bmp.scaleForMaxWidth(maxWidth)
try {
val out = ByteArrayOutputStream()
scaled.compress(Bitmap.CompressFormat.PNG, 100, out)
Base64.encodeToString(out.toByteArray(), Base64.NO_WRAP)
} finally {
if (scaled !== bmp) scaled.recycle()
}
} finally {
bmp.recycle()
}
}
suspend fun snapshotBase64(format: SnapshotFormat, quality: Double?, maxWidth: Int?): String =
withContext(Dispatchers.Main) {
val wv = webView ?: throw IllegalStateException("no webview")
val bmp = wv.captureBitmap()
try {
val scaled = bmp.scaleForMaxWidth(maxWidth)
try {
val out = ByteArrayOutputStream()
val (compressFormat, compressQuality) =
when (format) {
SnapshotFormat.Png -> Bitmap.CompressFormat.PNG to 100
SnapshotFormat.Jpeg -> Bitmap.CompressFormat.JPEG to clampJpegQuality(quality)
}
scaled.compress(compressFormat, compressQuality, out)
Base64.encodeToString(out.toByteArray(), Base64.NO_WRAP)
} finally {
if (scaled !== bmp) scaled.recycle()
}
} finally {
bmp.recycle()
}
}
private suspend fun WebView.captureBitmap(): Bitmap =
suspendCancellableCoroutine { cont ->
val width = width.coerceAtLeast(1)
val height = height.coerceAtLeast(1)
val bitmap = createBitmap(width, height, Bitmap.Config.ARGB_8888)
// WebView isn't supported by PixelCopy.request(...) directly; draw() is the most reliable
// cross-version snapshot for this lightweight "canvas" use-case.
draw(Canvas(bitmap))
cont.resume(bitmap)
}
companion object {
data class SnapshotParams(val format: SnapshotFormat, val quality: Double?, val maxWidth: Int?)
fun parseNavigateUrl(paramsJson: String?): String {
val obj = parseParamsObject(paramsJson) ?: return ""
return obj.string("url").trim()
}
fun parseEvalJs(paramsJson: String?): String? {
val obj = parseParamsObject(paramsJson) ?: return null
val js = obj.string("javaScript").trim()
return js.takeIf { it.isNotBlank() }
}
fun parseSnapshotMaxWidth(paramsJson: String?): Int? {
val obj = parseParamsObject(paramsJson) ?: return null
if (!obj.containsKey("maxWidth")) return null
val width = obj.int("maxWidth") ?: 0
return width.takeIf { it > 0 }
}
fun parseSnapshotFormat(paramsJson: String?): SnapshotFormat {
val obj = parseParamsObject(paramsJson) ?: return SnapshotFormat.Jpeg
val raw = obj.string("format").trim().lowercase()
return when (raw) {
"png" -> SnapshotFormat.Png
"jpeg", "jpg" -> SnapshotFormat.Jpeg
"" -> SnapshotFormat.Jpeg
else -> SnapshotFormat.Jpeg
}
}
fun parseSnapshotQuality(paramsJson: String?): Double? {
val obj = parseParamsObject(paramsJson) ?: return null
if (!obj.containsKey("quality")) return null
val q = obj.double("quality") ?: Double.NaN
if (!q.isFinite()) return null
return q.coerceIn(0.1, 1.0)
}
fun parseSnapshotParams(paramsJson: String?): SnapshotParams {
return SnapshotParams(
format = parseSnapshotFormat(paramsJson),
quality = parseSnapshotQuality(paramsJson),
maxWidth = parseSnapshotMaxWidth(paramsJson),
)
}
private val json = Json { ignoreUnknownKeys = true }
private fun parseParamsObject(paramsJson: String?): JsonObject? {
val raw = paramsJson?.trim().orEmpty()
if (raw.isEmpty()) return null
return try {
json.parseToJsonElement(raw).asObjectOrNull()
} catch (_: Throwable) {
null
}
}
private fun JsonElement?.asObjectOrNull(): JsonObject? = this as? JsonObject
private fun JsonObject.string(key: String): String {
val prim = this[key] as? JsonPrimitive ?: return ""
val raw = prim.content
return raw.takeIf { it != "null" }.orEmpty()
}
private fun JsonObject.int(key: String): Int? {
val prim = this[key] as? JsonPrimitive ?: return null
return prim.content.toIntOrNull()
}
private fun JsonObject.double(key: String): Double? {
val prim = this[key] as? JsonPrimitive ?: return null
return prim.content.toDoubleOrNull()
}
}
}

View file

@ -0,0 +1,173 @@
package ai.openclaw.app.node
import android.os.Build
import ai.openclaw.app.BuildConfig
import ai.openclaw.app.SecurePrefs
import ai.openclaw.app.gateway.GatewayClientInfo
import ai.openclaw.app.gateway.GatewayConnectOptions
import ai.openclaw.app.gateway.GatewayEndpoint
import ai.openclaw.app.gateway.GatewayTlsParams
import ai.openclaw.app.gateway.isPrivateLanGatewayHost
import ai.openclaw.app.LocationMode
import ai.openclaw.app.VoiceWakeMode
class ConnectionManager(
private val prefs: SecurePrefs,
private val cameraEnabled: () -> Boolean,
private val locationMode: () -> LocationMode,
private val voiceWakeMode: () -> VoiceWakeMode,
private val motionActivityAvailable: () -> Boolean,
private val motionPedometerAvailable: () -> Boolean,
private val sendSmsAvailable: () -> Boolean,
private val readSmsAvailable: () -> Boolean,
private val smsSearchPossible: () -> Boolean,
private val callLogAvailable: () -> Boolean,
private val hasRecordAudioPermission: () -> Boolean,
private val manualTls: () -> Boolean,
) {
companion object {
internal fun resolveTlsParamsForEndpoint(
endpoint: GatewayEndpoint,
storedFingerprint: String?,
manualTlsEnabled: Boolean,
): GatewayTlsParams? {
val stableId = endpoint.stableId
val stored = storedFingerprint?.trim().takeIf { !it.isNullOrEmpty() }
val isManual = stableId.startsWith("manual|")
val cleartextAllowedHost = isPrivateLanGatewayHost(endpoint.host)
if (isManual) {
if (!manualTlsEnabled && cleartextAllowedHost) return null
if (!stored.isNullOrBlank()) {
return GatewayTlsParams(
required = true,
expectedFingerprint = stored,
allowTOFU = false,
stableId = stableId,
)
}
return GatewayTlsParams(
required = true,
expectedFingerprint = null,
allowTOFU = false,
stableId = stableId,
)
}
// Prefer stored pins. Never let discovery-provided TXT override a stored fingerprint.
if (!stored.isNullOrBlank()) {
return GatewayTlsParams(
required = true,
expectedFingerprint = stored,
allowTOFU = false,
stableId = stableId,
)
}
val hinted = endpoint.tlsEnabled || !endpoint.tlsFingerprintSha256.isNullOrBlank()
if (hinted) {
// TXT is unauthenticated. Do not treat the advertised fingerprint as authoritative.
return GatewayTlsParams(
required = true,
expectedFingerprint = null,
allowTOFU = false,
stableId = stableId,
)
}
if (!cleartextAllowedHost) {
return GatewayTlsParams(
required = true,
expectedFingerprint = null,
allowTOFU = false,
stableId = stableId,
)
}
return null
}
}
private fun runtimeFlags(): NodeRuntimeFlags =
NodeRuntimeFlags(
cameraEnabled = cameraEnabled(),
locationEnabled = locationMode() != LocationMode.Off,
sendSmsAvailable = sendSmsAvailable(),
readSmsAvailable = readSmsAvailable(),
smsSearchPossible = smsSearchPossible(),
callLogAvailable = callLogAvailable(),
voiceWakeEnabled = voiceWakeMode() != VoiceWakeMode.Off && hasRecordAudioPermission(),
motionActivityAvailable = motionActivityAvailable(),
motionPedometerAvailable = motionPedometerAvailable(),
debugBuild = BuildConfig.DEBUG,
)
fun buildInvokeCommands(): List<String> = InvokeCommandRegistry.advertisedCommands(runtimeFlags())
fun buildCapabilities(): List<String> = InvokeCommandRegistry.advertisedCapabilities(runtimeFlags())
fun resolvedVersionName(): String {
val versionName = BuildConfig.VERSION_NAME.trim().ifEmpty { "dev" }
return if (BuildConfig.DEBUG && !versionName.contains("dev", ignoreCase = true)) {
"$versionName-dev"
} else {
versionName
}
}
fun resolveModelIdentifier(): String? {
return listOfNotNull(Build.MANUFACTURER, Build.MODEL)
.joinToString(" ")
.trim()
.ifEmpty { null }
}
fun buildUserAgent(): String {
val version = resolvedVersionName()
val release = Build.VERSION.RELEASE?.trim().orEmpty()
val releaseLabel = if (release.isEmpty()) "unknown" else release
return "OpenClawAndroid/$version (Android $releaseLabel; SDK ${Build.VERSION.SDK_INT})"
}
fun buildClientInfo(clientId: String, clientMode: String): GatewayClientInfo {
return GatewayClientInfo(
id = clientId,
displayName = prefs.displayName.value,
version = resolvedVersionName(),
platform = "android",
mode = clientMode,
instanceId = prefs.instanceId.value,
deviceFamily = "Android",
modelIdentifier = resolveModelIdentifier(),
)
}
fun buildNodeConnectOptions(): GatewayConnectOptions {
return GatewayConnectOptions(
role = "node",
scopes = emptyList(),
caps = buildCapabilities(),
commands = buildInvokeCommands(),
permissions = emptyMap(),
client = buildClientInfo(clientId = "openclaw-android", clientMode = "node"),
userAgent = buildUserAgent(),
)
}
fun buildOperatorConnectOptions(): GatewayConnectOptions {
return GatewayConnectOptions(
role = "operator",
scopes = listOf("operator.read", "operator.write", "operator.talk.secrets"),
caps = emptyList(),
commands = emptyList(),
permissions = emptyMap(),
client = buildClientInfo(clientId = "openclaw-android", clientMode = "ui"),
userAgent = buildUserAgent(),
)
}
fun resolveTlsParams(endpoint: GatewayEndpoint): GatewayTlsParams? {
val stored = prefs.loadGatewayTlsFingerprint(endpoint.stableId)
return resolveTlsParamsForEndpoint(endpoint, storedFingerprint = stored, manualTlsEnabled = manualTls())
}
}

View file

@ -0,0 +1,433 @@
package ai.openclaw.app.node
import android.Manifest
import android.content.ContentProviderOperation
import android.content.ContentResolver
import android.content.ContentValues
import android.content.Context
import android.provider.ContactsContract
import androidx.core.content.ContextCompat
import ai.openclaw.app.gateway.GatewaySession
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonArray
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.buildJsonArray
import kotlinx.serialization.json.buildJsonObject
import kotlinx.serialization.json.put
private const val DEFAULT_CONTACTS_LIMIT = 25
internal data class ContactRecord(
val identifier: String,
val displayName: String,
val givenName: String,
val familyName: String,
val organizationName: String,
val phoneNumbers: List<String>,
val emails: List<String>,
)
internal data class ContactsSearchRequest(
val query: String?,
val limit: Int,
)
internal data class ContactsAddRequest(
val givenName: String?,
val familyName: String?,
val organizationName: String?,
val displayName: String?,
val phoneNumbers: List<String>,
val emails: List<String>,
)
internal interface ContactsDataSource {
fun hasReadPermission(context: Context): Boolean
fun hasWritePermission(context: Context): Boolean
fun search(context: Context, request: ContactsSearchRequest): List<ContactRecord>
fun add(context: Context, request: ContactsAddRequest): ContactRecord
}
private object SystemContactsDataSource : ContactsDataSource {
override fun hasReadPermission(context: Context): Boolean {
return ContextCompat.checkSelfPermission(context, Manifest.permission.READ_CONTACTS) ==
android.content.pm.PackageManager.PERMISSION_GRANTED
}
override fun hasWritePermission(context: Context): Boolean {
return ContextCompat.checkSelfPermission(context, Manifest.permission.WRITE_CONTACTS) ==
android.content.pm.PackageManager.PERMISSION_GRANTED
}
override fun search(context: Context, request: ContactsSearchRequest): List<ContactRecord> {
val resolver = context.contentResolver
val projection =
arrayOf(
ContactsContract.Contacts._ID,
ContactsContract.Contacts.DISPLAY_NAME_PRIMARY,
)
val selection: String?
val selectionArgs: Array<String>?
if (request.query.isNullOrBlank()) {
selection = null
selectionArgs = null
} else {
selection = "${ContactsContract.Contacts.DISPLAY_NAME_PRIMARY} LIKE ? ESCAPE '\\'"
selectionArgs = arrayOf("%${escapeLikePattern(request.query)}%")
}
val sortOrder = "${ContactsContract.Contacts.DISPLAY_NAME_PRIMARY} COLLATE NOCASE ASC LIMIT ${request.limit}"
resolver.query(
ContactsContract.Contacts.CONTENT_URI,
projection,
selection,
selectionArgs,
sortOrder,
).use { cursor ->
if (cursor == null) return emptyList()
val idIndex = cursor.getColumnIndexOrThrow(ContactsContract.Contacts._ID)
val displayNameIndex = cursor.getColumnIndexOrThrow(ContactsContract.Contacts.DISPLAY_NAME_PRIMARY)
val out = mutableListOf<ContactRecord>()
while (cursor.moveToNext() && out.size < request.limit) {
val contactId = cursor.getLong(idIndex)
val displayName = cursor.getString(displayNameIndex).orEmpty()
out += loadContactRecord(resolver, contactId, fallbackDisplayName = displayName)
}
return out
}
}
override fun add(context: Context, request: ContactsAddRequest): ContactRecord {
val resolver = context.contentResolver
val operations = ArrayList<ContentProviderOperation>()
operations +=
ContentProviderOperation.newInsert(ContactsContract.RawContacts.CONTENT_URI)
.withValue(ContactsContract.RawContacts.ACCOUNT_TYPE, null)
.withValue(ContactsContract.RawContacts.ACCOUNT_NAME, null)
.build()
if (!request.givenName.isNullOrEmpty() || !request.familyName.isNullOrEmpty() || !request.displayName.isNullOrEmpty()) {
operations +=
ContentProviderOperation.newInsert(ContactsContract.Data.CONTENT_URI)
.withValueBackReference(ContactsContract.Data.RAW_CONTACT_ID, 0)
.withValue(ContactsContract.Data.MIMETYPE, ContactsContract.CommonDataKinds.StructuredName.CONTENT_ITEM_TYPE)
.withValue(ContactsContract.CommonDataKinds.StructuredName.GIVEN_NAME, request.givenName)
.withValue(ContactsContract.CommonDataKinds.StructuredName.FAMILY_NAME, request.familyName)
.withValue(ContactsContract.CommonDataKinds.StructuredName.DISPLAY_NAME, request.displayName)
.build()
}
if (!request.organizationName.isNullOrEmpty()) {
operations +=
ContentProviderOperation.newInsert(ContactsContract.Data.CONTENT_URI)
.withValueBackReference(ContactsContract.Data.RAW_CONTACT_ID, 0)
.withValue(ContactsContract.Data.MIMETYPE, ContactsContract.CommonDataKinds.Organization.CONTENT_ITEM_TYPE)
.withValue(ContactsContract.CommonDataKinds.Organization.COMPANY, request.organizationName)
.build()
}
request.phoneNumbers.forEach { number ->
operations +=
ContentProviderOperation.newInsert(ContactsContract.Data.CONTENT_URI)
.withValueBackReference(ContactsContract.Data.RAW_CONTACT_ID, 0)
.withValue(ContactsContract.Data.MIMETYPE, ContactsContract.CommonDataKinds.Phone.CONTENT_ITEM_TYPE)
.withValue(ContactsContract.CommonDataKinds.Phone.NUMBER, number)
.withValue(ContactsContract.CommonDataKinds.Phone.TYPE, ContactsContract.CommonDataKinds.Phone.TYPE_MOBILE)
.build()
}
request.emails.forEach { email ->
operations +=
ContentProviderOperation.newInsert(ContactsContract.Data.CONTENT_URI)
.withValueBackReference(ContactsContract.Data.RAW_CONTACT_ID, 0)
.withValue(ContactsContract.Data.MIMETYPE, ContactsContract.CommonDataKinds.Email.CONTENT_ITEM_TYPE)
.withValue(ContactsContract.CommonDataKinds.Email.ADDRESS, email)
.withValue(ContactsContract.CommonDataKinds.Email.TYPE, ContactsContract.CommonDataKinds.Email.TYPE_HOME)
.build()
}
val results = resolver.applyBatch(ContactsContract.AUTHORITY, operations)
val rawContactUri = results.firstOrNull()?.uri
?: throw IllegalStateException("contact insert failed")
val rawContactId = rawContactUri.lastPathSegment?.toLongOrNull()
?: throw IllegalStateException("contact insert failed")
val contactId = resolveContactIdForRawContact(resolver, rawContactId)
?: throw IllegalStateException("contact insert failed")
return loadContactRecord(
resolver = resolver,
contactId = contactId,
fallbackDisplayName = request.displayName.orEmpty(),
)
}
private fun resolveContactIdForRawContact(resolver: ContentResolver, rawContactId: Long): Long? {
val projection = arrayOf(ContactsContract.RawContacts.CONTACT_ID)
resolver.query(
ContactsContract.RawContacts.CONTENT_URI,
projection,
"${ContactsContract.RawContacts._ID}=?",
arrayOf(rawContactId.toString()),
null,
).use { cursor ->
if (cursor == null || !cursor.moveToFirst()) return null
val index = cursor.getColumnIndexOrThrow(ContactsContract.RawContacts.CONTACT_ID)
return cursor.getLong(index)
}
}
private fun loadContactRecord(
resolver: ContentResolver,
contactId: Long,
fallbackDisplayName: String,
): ContactRecord {
val nameRow = loadNameRow(resolver, contactId)
val organization = loadOrganization(resolver, contactId)
val phones = loadPhones(resolver, contactId)
val emails = loadEmails(resolver, contactId)
val displayName =
when {
!nameRow.displayName.isNullOrEmpty() -> nameRow.displayName
!fallbackDisplayName.isNullOrEmpty() -> fallbackDisplayName
else -> listOfNotNull(nameRow.givenName, nameRow.familyName).joinToString(" ").trim()
}.ifEmpty { "(unnamed)" }
return ContactRecord(
identifier = contactId.toString(),
displayName = displayName,
givenName = nameRow.givenName.orEmpty(),
familyName = nameRow.familyName.orEmpty(),
organizationName = organization.orEmpty(),
phoneNumbers = phones,
emails = emails,
)
}
private data class NameRow(
val givenName: String?,
val familyName: String?,
val displayName: String?,
)
private fun loadNameRow(resolver: ContentResolver, contactId: Long): NameRow {
val projection =
arrayOf(
ContactsContract.CommonDataKinds.StructuredName.GIVEN_NAME,
ContactsContract.CommonDataKinds.StructuredName.FAMILY_NAME,
ContactsContract.CommonDataKinds.StructuredName.DISPLAY_NAME,
)
resolver.query(
ContactsContract.Data.CONTENT_URI,
projection,
"${ContactsContract.Data.CONTACT_ID}=? AND ${ContactsContract.Data.MIMETYPE}=?",
arrayOf(
contactId.toString(),
ContactsContract.CommonDataKinds.StructuredName.CONTENT_ITEM_TYPE,
),
null,
).use { cursor ->
if (cursor == null || !cursor.moveToFirst()) {
return NameRow(givenName = null, familyName = null, displayName = null)
}
val given = cursor.getString(0)?.trim()?.ifEmpty { null }
val family = cursor.getString(1)?.trim()?.ifEmpty { null }
val display = cursor.getString(2)?.trim()?.ifEmpty { null }
return NameRow(givenName = given, familyName = family, displayName = display)
}
}
private fun loadOrganization(resolver: ContentResolver, contactId: Long): String? {
val projection = arrayOf(ContactsContract.CommonDataKinds.Organization.COMPANY)
resolver.query(
ContactsContract.Data.CONTENT_URI,
projection,
"${ContactsContract.Data.CONTACT_ID}=? AND ${ContactsContract.Data.MIMETYPE}=?",
arrayOf(contactId.toString(), ContactsContract.CommonDataKinds.Organization.CONTENT_ITEM_TYPE),
null,
).use { cursor ->
if (cursor == null || !cursor.moveToFirst()) return null
return cursor.getString(0)?.trim()?.ifEmpty { null }
}
}
private fun escapeLikePattern(pattern: String): String =
pattern.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
private fun loadPhones(resolver: ContentResolver, contactId: Long): List<String> {
return queryContactValues(
resolver = resolver,
contentUri = ContactsContract.CommonDataKinds.Phone.CONTENT_URI,
valueColumn = ContactsContract.CommonDataKinds.Phone.NUMBER,
contactIdColumn = ContactsContract.CommonDataKinds.Phone.CONTACT_ID,
contactId = contactId,
)
}
private fun loadEmails(resolver: ContentResolver, contactId: Long): List<String> {
return queryContactValues(
resolver = resolver,
contentUri = ContactsContract.CommonDataKinds.Email.CONTENT_URI,
valueColumn = ContactsContract.CommonDataKinds.Email.ADDRESS,
contactIdColumn = ContactsContract.CommonDataKinds.Email.CONTACT_ID,
contactId = contactId,
)
}
private fun queryContactValues(
resolver: ContentResolver,
contentUri: android.net.Uri,
valueColumn: String,
contactIdColumn: String,
contactId: Long,
): List<String> {
val projection = arrayOf(valueColumn)
resolver.query(
contentUri,
projection,
"$contactIdColumn=?",
arrayOf(contactId.toString()),
null,
).use { cursor ->
if (cursor == null) return emptyList()
val out = LinkedHashSet<String>()
while (cursor.moveToNext()) {
val value = cursor.getString(0)?.trim().orEmpty()
if (value.isNotEmpty()) out += value
}
return out.toList()
}
}
}
class ContactsHandler private constructor(
private val appContext: Context,
private val dataSource: ContactsDataSource,
) {
constructor(appContext: Context) : this(appContext = appContext, dataSource = SystemContactsDataSource)
fun handleContactsSearch(paramsJson: String?): GatewaySession.InvokeResult {
if (!dataSource.hasReadPermission(appContext)) {
return GatewaySession.InvokeResult.error(
code = "CONTACTS_PERMISSION_REQUIRED",
message = "CONTACTS_PERMISSION_REQUIRED: grant Contacts permission",
)
}
val request =
parseSearchRequest(paramsJson)
?: return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: expected JSON object",
)
return try {
val contacts = dataSource.search(appContext, request)
GatewaySession.InvokeResult.ok(
buildJsonObject {
put(
"contacts",
buildJsonArray {
contacts.forEach { add(contactJson(it)) }
},
)
}.toString(),
)
} catch (err: Throwable) {
GatewaySession.InvokeResult.error(
code = "CONTACTS_UNAVAILABLE",
message = "CONTACTS_UNAVAILABLE: ${err.message ?: "contacts query failed"}",
)
}
}
fun handleContactsAdd(paramsJson: String?): GatewaySession.InvokeResult {
if (!dataSource.hasWritePermission(appContext)) {
return GatewaySession.InvokeResult.error(
code = "CONTACTS_PERMISSION_REQUIRED",
message = "CONTACTS_PERMISSION_REQUIRED: grant Contacts permission",
)
}
val request =
parseAddRequest(paramsJson)
?: return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: expected JSON object",
)
val hasName =
!(request.givenName.isNullOrEmpty() && request.familyName.isNullOrEmpty() && request.displayName.isNullOrEmpty())
val hasOrg = !request.organizationName.isNullOrEmpty()
val hasDetails = request.phoneNumbers.isNotEmpty() || request.emails.isNotEmpty()
if (!hasName && !hasOrg && !hasDetails) {
return GatewaySession.InvokeResult.error(
code = "CONTACTS_INVALID",
message = "CONTACTS_INVALID: include a name, organization, phone, or email",
)
}
return try {
val contact = dataSource.add(appContext, request)
GatewaySession.InvokeResult.ok(
buildJsonObject {
put("contact", contactJson(contact))
}.toString(),
)
} catch (err: Throwable) {
GatewaySession.InvokeResult.error(
code = "CONTACTS_UNAVAILABLE",
message = "CONTACTS_UNAVAILABLE: ${err.message ?: "contact add failed"}",
)
}
}
private fun parseSearchRequest(paramsJson: String?): ContactsSearchRequest? {
if (paramsJson.isNullOrBlank()) {
return ContactsSearchRequest(query = null, limit = DEFAULT_CONTACTS_LIMIT)
}
val params =
try {
Json.parseToJsonElement(paramsJson).asObjectOrNull()
} catch (_: Throwable) {
null
} ?: return null
val query = (params["query"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null }
val limit = ((params["limit"] as? JsonPrimitive)?.content?.toIntOrNull() ?: DEFAULT_CONTACTS_LIMIT).coerceIn(1, 200)
return ContactsSearchRequest(query = query, limit = limit)
}
private fun parseAddRequest(paramsJson: String?): ContactsAddRequest? {
val params =
try {
paramsJson?.let { Json.parseToJsonElement(it).asObjectOrNull() }
} catch (_: Throwable) {
null
} ?: return null
return ContactsAddRequest(
givenName = (params["givenName"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
familyName = (params["familyName"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
organizationName = (params["organizationName"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
displayName = (params["displayName"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
phoneNumbers = stringArray(params["phoneNumbers"] as? JsonArray),
emails = stringArray(params["emails"] as? JsonArray).map { it.lowercase() },
)
}
private fun stringArray(array: JsonArray?): List<String> {
if (array == null) return emptyList()
return array.mapNotNull { element ->
(element as? JsonPrimitive)?.content?.trim()?.ifEmpty { null }
}
}
private fun contactJson(contact: ContactRecord): JsonObject {
return buildJsonObject {
put("identifier", JsonPrimitive(contact.identifier))
put("displayName", JsonPrimitive(contact.displayName))
put("givenName", JsonPrimitive(contact.givenName))
put("familyName", JsonPrimitive(contact.familyName))
put("organizationName", JsonPrimitive(contact.organizationName))
put("phoneNumbers", buildJsonArray { contact.phoneNumbers.forEach { add(JsonPrimitive(it)) } })
put("emails", buildJsonArray { contact.emails.forEach { add(JsonPrimitive(it)) } })
}
}
companion object {
internal fun forTesting(
appContext: Context,
dataSource: ContactsDataSource,
): ContactsHandler = ContactsHandler(appContext = appContext, dataSource = dataSource)
}
}

View file

@ -0,0 +1,118 @@
package ai.openclaw.app.node
import android.content.Context
import ai.openclaw.app.BuildConfig
import ai.openclaw.app.gateway.DeviceIdentityStore
import ai.openclaw.app.gateway.GatewaySession
import kotlinx.serialization.json.JsonPrimitive
class DebugHandler(
private val appContext: Context,
private val identityStore: DeviceIdentityStore,
) {
fun handleEd25519(): GatewaySession.InvokeResult {
if (!BuildConfig.DEBUG) {
return GatewaySession.InvokeResult.error(code = "UNAVAILABLE", message = "debug commands are disabled in release builds")
}
// Self-test Ed25519 signing and return diagnostic info
try {
val identity = identityStore.loadOrCreate()
val testPayload = "test|${identity.deviceId}|${System.currentTimeMillis()}"
val results = mutableListOf<String>()
results.add("deviceId: ${identity.deviceId}")
results.add("publicKeyRawBase64: ${identity.publicKeyRawBase64.take(20)}...")
results.add("privateKeyPkcs8Base64: ${identity.privateKeyPkcs8Base64.take(20)}...")
// Test publicKeyBase64Url
val pubKeyUrl = identityStore.publicKeyBase64Url(identity)
results.add("publicKeyBase64Url: ${pubKeyUrl ?: "NULL (FAILED)"}")
// Test signing
val signature = identityStore.signPayload(testPayload, identity)
results.add("signPayload: ${if (signature != null) "${signature.take(20)}... (OK)" else "NULL (FAILED)"}")
// Test self-verify
if (signature != null) {
val verifyOk = identityStore.verifySelfSignature(testPayload, signature, identity)
results.add("verifySelfSignature: $verifyOk")
}
// Check available providers
val providers = java.security.Security.getProviders()
val ed25519Providers = providers.filter { p ->
p.services.any { s -> s.algorithm.contains("Ed25519", ignoreCase = true) }
}
results.add("Ed25519 providers: ${ed25519Providers.map { "${it.name} v${it.version}" }}")
results.add("Provider order: ${providers.take(5).map { it.name }}")
// Test KeyFactory directly
try {
val kf = java.security.KeyFactory.getInstance("Ed25519")
results.add("KeyFactory.Ed25519: ${kf.provider.name} (OK)")
} catch (e: Throwable) {
results.add("KeyFactory.Ed25519: FAILED - ${e.javaClass.simpleName}: ${e.message}")
}
// Test Signature directly
try {
val sig = java.security.Signature.getInstance("Ed25519")
results.add("Signature.Ed25519: ${sig.provider.name} (OK)")
} catch (e: Throwable) {
results.add("Signature.Ed25519: FAILED - ${e.javaClass.simpleName}: ${e.message}")
}
val diagnostics = results.joinToString("\n")
return GatewaySession.InvokeResult.ok("""{"diagnostics":${JsonPrimitive(diagnostics)}}""")
} catch (e: Throwable) {
return GatewaySession.InvokeResult.error(code = "ED25519_TEST_FAILED", message = "${e.javaClass.simpleName}: ${e.message}\n${e.stackTraceToString().take(500)}")
}
}
fun handleLogs(): GatewaySession.InvokeResult {
if (!BuildConfig.DEBUG) {
return GatewaySession.InvokeResult.error(code = "UNAVAILABLE", message = "debug commands are disabled in release builds")
}
val pid = android.os.Process.myPid()
val rt = Runtime.getRuntime()
val info = "v6 pid=$pid thread=${Thread.currentThread().name} free=${rt.freeMemory()/1024}K total=${rt.totalMemory()/1024}K max=${rt.maxMemory()/1024}K uptime=${android.os.SystemClock.elapsedRealtime()/1000}s sdk=${android.os.Build.VERSION.SDK_INT} device=${android.os.Build.MODEL}\n"
// Run logcat on current dispatcher thread (no withContext) with file redirect
val logResult = try {
val tmpFile = java.io.File(appContext.cacheDir, "debug_logs.txt")
if (tmpFile.exists()) tmpFile.delete()
val pb = ProcessBuilder("logcat", "-d", "-t", "200", "--pid=$pid")
pb.redirectOutput(tmpFile)
pb.redirectErrorStream(true)
val proc = pb.start()
val finished = proc.waitFor(4, java.util.concurrent.TimeUnit.SECONDS)
if (!finished) proc.destroyForcibly()
val raw = if (tmpFile.exists() && tmpFile.length() > 0) {
tmpFile.readText().take(128000)
} else {
"(no output, finished=$finished, exists=${tmpFile.exists()})"
}
tmpFile.delete()
val spamPatterns = listOf("setRequestedFrameRate", "I View :", "BLASTBufferQueue", "VRI[Pop-Up",
"InsetsController:", "VRI[MainActivity", "InsetsSource:", "handleResized", "ProfileInstaller",
"I VRI[", "onStateChanged: host=", "D StrictMode:", "E StrictMode:", "ImeFocusController",
"InputTransport", "IncorrectContextUseViolation")
val sb = StringBuilder()
for (line in raw.lineSequence()) {
if (line.isBlank()) continue
if (spamPatterns.any { line.contains(it) }) continue
if (sb.length + line.length > 16000) { sb.append("\n(truncated)"); break }
if (sb.isNotEmpty()) sb.append('\n')
sb.append(line)
}
sb.toString().ifEmpty { "(all ${raw.lines().size} lines filtered as spam)" }
} catch (e: Throwable) {
"(logcat error: ${e::class.java.simpleName}: ${e.message})"
}
// Also include camera debug log if it exists
val camLogFile = java.io.File(appContext.cacheDir, "camera_debug.log")
val camLog = if (camLogFile.exists() && camLogFile.length() > 0) {
"\n--- camera_debug.log ---\n" + camLogFile.readText().take(4000)
} else ""
return GatewaySession.InvokeResult.ok("""{"logs":${JsonPrimitive(info + logResult + camLog)}}""")
}
}

View file

@ -0,0 +1,453 @@
package ai.openclaw.app.node
import ai.openclaw.app.BuildConfig
import android.Manifest
import android.app.ActivityManager
import android.content.Context
import android.content.Intent
import android.content.IntentFilter
import android.content.pm.PackageManager
import android.net.ConnectivityManager
import android.net.NetworkCapabilities
import android.os.BatteryManager
import android.os.Build
import android.os.Environment
import android.os.PowerManager
import android.os.StatFs
import android.os.SystemClock
import androidx.core.content.ContextCompat
import ai.openclaw.app.gateway.GatewaySession
import java.util.Locale
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.buildJsonArray
import kotlinx.serialization.json.buildJsonObject
import kotlinx.serialization.json.put
class DeviceHandler(
private val appContext: Context,
private val smsEnabled: Boolean = BuildConfig.OPENCLAW_ENABLE_SMS,
private val callLogEnabled: Boolean = BuildConfig.OPENCLAW_ENABLE_CALL_LOG,
) {
companion object {
internal fun hasAnySmsCapability(
smsEnabled: Boolean,
telephonyAvailable: Boolean,
smsSendGranted: Boolean,
smsReadGranted: Boolean,
): Boolean {
return smsEnabled && telephonyAvailable && (smsSendGranted || smsReadGranted)
}
internal fun isSmsPromptable(
smsEnabled: Boolean,
telephonyAvailable: Boolean,
smsSendGranted: Boolean,
smsReadGranted: Boolean,
): Boolean {
return smsEnabled && telephonyAvailable && (!smsSendGranted || !smsReadGranted)
}
}
private data class BatterySnapshot(
val status: Int,
val plugged: Int,
val levelFraction: Double?,
val temperatureC: Double?,
)
fun handleDeviceStatus(_paramsJson: String?): GatewaySession.InvokeResult {
return GatewaySession.InvokeResult.ok(statusPayloadJson())
}
fun handleDeviceInfo(_paramsJson: String?): GatewaySession.InvokeResult {
return GatewaySession.InvokeResult.ok(infoPayloadJson())
}
fun handleDevicePermissions(_paramsJson: String?): GatewaySession.InvokeResult {
return GatewaySession.InvokeResult.ok(permissionsPayloadJson())
}
fun handleDeviceHealth(_paramsJson: String?): GatewaySession.InvokeResult {
return GatewaySession.InvokeResult.ok(healthPayloadJson())
}
private fun statusPayloadJson(): String {
val battery = readBatterySnapshot()
val powerManager = appContext.getSystemService(PowerManager::class.java)
val storage = StatFs(Environment.getDataDirectory().absolutePath)
val totalBytes = storage.totalBytes
val freeBytes = storage.availableBytes
val usedBytes = (totalBytes - freeBytes).coerceAtLeast(0L)
val connectivity = appContext.getSystemService(ConnectivityManager::class.java)
val activeNetwork = connectivity?.activeNetwork
val caps = activeNetwork?.let { connectivity.getNetworkCapabilities(it) }
val uptimeSeconds = SystemClock.elapsedRealtime() / 1_000.0
return buildJsonObject {
put(
"battery",
buildJsonObject {
battery.levelFraction?.let { put("level", JsonPrimitive(it)) }
put("state", JsonPrimitive(mapBatteryState(battery.status)))
put("lowPowerModeEnabled", JsonPrimitive(powerManager?.isPowerSaveMode == true))
},
)
put(
"thermal",
buildJsonObject {
put("state", JsonPrimitive(mapThermalState(powerManager)))
},
)
put(
"storage",
buildJsonObject {
put("totalBytes", JsonPrimitive(totalBytes))
put("freeBytes", JsonPrimitive(freeBytes))
put("usedBytes", JsonPrimitive(usedBytes))
},
)
put(
"network",
buildJsonObject {
put("status", JsonPrimitive(mapNetworkStatus(caps)))
put(
"isExpensive",
JsonPrimitive(
caps?.hasCapability(NetworkCapabilities.NET_CAPABILITY_NOT_METERED)?.not() ?: false,
),
)
put(
"isConstrained",
JsonPrimitive(
caps?.hasCapability(NetworkCapabilities.NET_CAPABILITY_NOT_RESTRICTED)?.not() ?: false,
),
)
put("interfaces", networkInterfacesJson(caps))
},
)
put("uptimeSeconds", JsonPrimitive(uptimeSeconds))
}.toString()
}
private fun infoPayloadJson(): String {
val model = Build.MODEL?.trim().orEmpty()
val manufacturer = Build.MANUFACTURER?.trim().orEmpty()
val modelIdentifier = Build.DEVICE?.trim().orEmpty()
val systemVersion = Build.VERSION.RELEASE?.trim().orEmpty()
val locale = Locale.getDefault().toLanguageTag().trim()
val appVersion = BuildConfig.VERSION_NAME.trim()
val appBuild = BuildConfig.VERSION_CODE.toString()
return buildJsonObject {
put("deviceName", JsonPrimitive(model.ifEmpty { "Android" }))
put("modelIdentifier", JsonPrimitive(modelIdentifier.ifEmpty { listOf(manufacturer, model).filter { it.isNotEmpty() }.joinToString(" ") }))
put("systemName", JsonPrimitive("Android"))
put("systemVersion", JsonPrimitive(systemVersion.ifEmpty { Build.VERSION.SDK_INT.toString() }))
put("appVersion", JsonPrimitive(appVersion.ifEmpty { "dev" }))
put("appBuild", JsonPrimitive(appBuild.ifEmpty { "0" }))
put("locale", JsonPrimitive(locale.ifEmpty { Locale.getDefault().toString() }))
}.toString()
}
private fun permissionsPayloadJson(): String {
val canSendSms = appContext.packageManager.hasSystemFeature(PackageManager.FEATURE_TELEPHONY)
val smsSendGranted = hasPermission(Manifest.permission.SEND_SMS)
val smsReadGranted = hasPermission(Manifest.permission.READ_SMS)
val notificationAccess = DeviceNotificationListenerService.isAccessEnabled(appContext)
val photosGranted =
if (Build.VERSION.SDK_INT >= 33) {
hasPermission(Manifest.permission.READ_MEDIA_IMAGES)
} else {
hasPermission(Manifest.permission.READ_EXTERNAL_STORAGE)
}
val motionGranted = hasPermission(Manifest.permission.ACTIVITY_RECOGNITION)
val notificationsGranted =
if (Build.VERSION.SDK_INT >= 33) {
hasPermission(Manifest.permission.POST_NOTIFICATIONS)
} else {
true
}
return buildJsonObject {
put(
"permissions",
buildJsonObject {
put(
"camera",
permissionStateJson(
granted = hasPermission(Manifest.permission.CAMERA),
promptableWhenDenied = true,
),
)
put(
"microphone",
permissionStateJson(
granted = hasPermission(Manifest.permission.RECORD_AUDIO),
promptableWhenDenied = true,
),
)
put(
"location",
permissionStateJson(
granted =
hasPermission(Manifest.permission.ACCESS_FINE_LOCATION) ||
hasPermission(Manifest.permission.ACCESS_COARSE_LOCATION),
promptableWhenDenied = true,
),
)
put(
"sms",
buildJsonObject {
put(
"status",
JsonPrimitive(
if (hasAnySmsCapability(smsEnabled, canSendSms, smsSendGranted, smsReadGranted)) "granted" else "denied",
),
)
put("promptable", JsonPrimitive(isSmsPromptable(smsEnabled, canSendSms, smsSendGranted, smsReadGranted)))
put(
"capabilities",
buildJsonObject {
put(
"send",
permissionStateJson(
granted = smsEnabled && smsSendGranted && canSendSms,
promptableWhenDenied = smsEnabled && canSendSms,
),
)
put(
"read",
permissionStateJson(
granted = smsEnabled && smsReadGranted && canSendSms,
promptableWhenDenied = smsEnabled && canSendSms,
),
)
},
)
},
)
put(
"notificationListener",
permissionStateJson(
granted = notificationAccess,
promptableWhenDenied = true,
),
)
put(
"notifications",
permissionStateJson(
granted = notificationsGranted,
promptableWhenDenied = true,
),
)
put(
"photos",
permissionStateJson(
granted = photosGranted,
promptableWhenDenied = true,
),
)
put(
"contacts",
permissionStateJson(
granted = hasPermission(Manifest.permission.READ_CONTACTS),
promptableWhenDenied = true,
),
)
put(
"calendar",
permissionStateJson(
granted = hasPermission(Manifest.permission.READ_CALENDAR),
promptableWhenDenied = true,
),
)
put(
"callLog",
permissionStateJson(
granted = callLogEnabled && hasPermission(Manifest.permission.READ_CALL_LOG),
promptableWhenDenied = callLogEnabled,
),
)
put(
"motion",
permissionStateJson(
granted = motionGranted,
promptableWhenDenied = true,
),
)
},
)
}.toString()
}
private fun healthPayloadJson(): String {
val battery = readBatterySnapshot()
val batteryManager = appContext.getSystemService(BatteryManager::class.java)
val currentNowUa = batteryManager?.getLongProperty(BatteryManager.BATTERY_PROPERTY_CURRENT_NOW)
val currentNowMa =
if (currentNowUa == null || currentNowUa == Long.MIN_VALUE) {
null
} else {
currentNowUa.toDouble() / 1_000.0
}
val powerManager = appContext.getSystemService(PowerManager::class.java)
val activityManager = appContext.getSystemService(ActivityManager::class.java)
val memoryInfo = ActivityManager.MemoryInfo()
activityManager?.getMemoryInfo(memoryInfo)
val totalRamBytes = memoryInfo.totalMem.coerceAtLeast(0L)
val availableRamBytes = memoryInfo.availMem.coerceAtLeast(0L)
val usedRamBytes = (totalRamBytes - availableRamBytes).coerceAtLeast(0L)
val lowMemory = memoryInfo.lowMemory
val memoryPressure = mapMemoryPressure(totalRamBytes, availableRamBytes, lowMemory)
return buildJsonObject {
put(
"memory",
buildJsonObject {
put("pressure", JsonPrimitive(memoryPressure))
put("totalRamBytes", JsonPrimitive(totalRamBytes))
put("availableRamBytes", JsonPrimitive(availableRamBytes))
put("usedRamBytes", JsonPrimitive(usedRamBytes))
put("thresholdBytes", JsonPrimitive(memoryInfo.threshold.coerceAtLeast(0L)))
put("lowMemory", JsonPrimitive(lowMemory))
},
)
put(
"battery",
buildJsonObject {
put("state", JsonPrimitive(mapBatteryState(battery.status)))
put("chargingType", JsonPrimitive(mapChargingType(battery.plugged)))
battery.temperatureC?.let { put("temperatureC", JsonPrimitive(it)) }
currentNowMa?.let { put("currentMa", JsonPrimitive(it)) }
},
)
put(
"power",
buildJsonObject {
put("dozeModeEnabled", JsonPrimitive(powerManager?.isDeviceIdleMode == true))
put("lowPowerModeEnabled", JsonPrimitive(powerManager?.isPowerSaveMode == true))
},
)
put(
"system",
buildJsonObject {
Build.VERSION.SECURITY_PATCH
?.trim()
?.takeIf { it.isNotEmpty() }
?.let { put("securityPatchLevel", JsonPrimitive(it)) }
},
)
}.toString()
}
private fun readBatterySnapshot(): BatterySnapshot {
val intent = appContext.registerReceiver(null, IntentFilter(Intent.ACTION_BATTERY_CHANGED))
val status =
intent?.getIntExtra(BatteryManager.EXTRA_STATUS, BatteryManager.BATTERY_STATUS_UNKNOWN)
?: BatteryManager.BATTERY_STATUS_UNKNOWN
val plugged = intent?.getIntExtra(BatteryManager.EXTRA_PLUGGED, 0) ?: 0
val temperatureC =
intent
?.getIntExtra(BatteryManager.EXTRA_TEMPERATURE, Int.MIN_VALUE)
?.takeIf { it != Int.MIN_VALUE }
?.toDouble()
?.div(10.0)
return BatterySnapshot(
status = status,
plugged = plugged,
levelFraction = batteryLevelFraction(intent),
temperatureC = temperatureC,
)
}
private fun batteryLevelFraction(intent: Intent?): Double? {
val rawLevel = intent?.getIntExtra(BatteryManager.EXTRA_LEVEL, -1) ?: -1
val rawScale = intent?.getIntExtra(BatteryManager.EXTRA_SCALE, -1) ?: -1
if (rawLevel < 0 || rawScale <= 0) return null
return rawLevel.toDouble() / rawScale.toDouble()
}
private fun mapBatteryState(status: Int): String {
return when (status) {
BatteryManager.BATTERY_STATUS_CHARGING -> "charging"
BatteryManager.BATTERY_STATUS_FULL -> "full"
BatteryManager.BATTERY_STATUS_DISCHARGING, BatteryManager.BATTERY_STATUS_NOT_CHARGING -> "unplugged"
else -> "unknown"
}
}
private fun mapChargingType(plugged: Int): String {
return when (plugged) {
BatteryManager.BATTERY_PLUGGED_AC -> "ac"
BatteryManager.BATTERY_PLUGGED_USB -> "usb"
BatteryManager.BATTERY_PLUGGED_WIRELESS -> "wireless"
BatteryManager.BATTERY_PLUGGED_DOCK -> "dock"
else -> "none"
}
}
private fun mapThermalState(powerManager: PowerManager?): String {
val thermal = powerManager?.currentThermalStatus ?: return "nominal"
return when (thermal) {
PowerManager.THERMAL_STATUS_NONE, PowerManager.THERMAL_STATUS_LIGHT -> "nominal"
PowerManager.THERMAL_STATUS_MODERATE -> "fair"
PowerManager.THERMAL_STATUS_SEVERE -> "serious"
PowerManager.THERMAL_STATUS_CRITICAL,
PowerManager.THERMAL_STATUS_EMERGENCY,
PowerManager.THERMAL_STATUS_SHUTDOWN -> "critical"
else -> "nominal"
}
}
private fun mapNetworkStatus(caps: NetworkCapabilities?): String {
if (caps == null) return "unsatisfied"
return when {
caps.hasCapability(NetworkCapabilities.NET_CAPABILITY_VALIDATED) -> "satisfied"
caps.hasCapability(NetworkCapabilities.NET_CAPABILITY_INTERNET) -> "requiresConnection"
else -> "unsatisfied"
}
}
private fun permissionStateJson(granted: Boolean, promptableWhenDenied: Boolean) =
buildJsonObject {
put("status", JsonPrimitive(if (granted) "granted" else "denied"))
put("promptable", JsonPrimitive(!granted && promptableWhenDenied))
}
private fun hasPermission(permission: String): Boolean {
return (
ContextCompat.checkSelfPermission(appContext, permission) == PackageManager.PERMISSION_GRANTED
)
}
private fun mapMemoryPressure(totalBytes: Long, availableBytes: Long, lowMemory: Boolean): String {
if (totalBytes <= 0L) return if (lowMemory) "critical" else "unknown"
if (lowMemory) return "critical"
val freeRatio = availableBytes.toDouble() / totalBytes.toDouble()
return when {
freeRatio <= 0.05 -> "critical"
freeRatio <= 0.15 -> "high"
freeRatio <= 0.30 -> "moderate"
else -> "normal"
}
}
private fun networkInterfacesJson(caps: NetworkCapabilities?) =
buildJsonArray {
if (caps == null) return@buildJsonArray
var hasKnownTransport = false
if (caps.hasTransport(NetworkCapabilities.TRANSPORT_WIFI)) {
hasKnownTransport = true
add(JsonPrimitive("wifi"))
}
if (caps.hasTransport(NetworkCapabilities.TRANSPORT_CELLULAR)) {
hasKnownTransport = true
add(JsonPrimitive("cellular"))
}
if (caps.hasTransport(NetworkCapabilities.TRANSPORT_ETHERNET)) {
hasKnownTransport = true
add(JsonPrimitive("wired"))
}
if (!hasKnownTransport) add(JsonPrimitive("other"))
}
}

View file

@ -0,0 +1,473 @@
package ai.openclaw.app.node
import android.app.Notification
import android.app.NotificationManager
import android.app.RemoteInput
import android.content.ComponentName
import android.content.Context
import android.content.Intent
import android.service.notification.NotificationListenerService
import android.service.notification.StatusBarNotification
import ai.openclaw.app.NotificationBurstLimiter
import ai.openclaw.app.SecurePrefs
import ai.openclaw.app.allowsPackage
import ai.openclaw.app.isWithinQuietHours
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.buildJsonObject
import kotlinx.serialization.json.put
private const val MAX_NOTIFICATION_TEXT_CHARS = 512
private const val NOTIFICATIONS_CHANGED_EVENT = "notifications.changed"
internal fun sanitizeNotificationText(value: CharSequence?): String? {
val normalized = value?.toString()?.trim().orEmpty()
return normalized.take(MAX_NOTIFICATION_TEXT_CHARS).ifEmpty { null }
}
data class DeviceNotificationEntry(
val key: String,
val packageName: String,
val title: String?,
val text: String?,
val subText: String?,
val category: String?,
val channelId: String?,
val postTimeMs: Long,
val isOngoing: Boolean,
val isClearable: Boolean,
)
internal fun DeviceNotificationEntry.toJsonObject(): JsonObject {
return buildJsonObject {
put("key", JsonPrimitive(key))
put("packageName", JsonPrimitive(packageName))
put("postTimeMs", JsonPrimitive(postTimeMs))
put("isOngoing", JsonPrimitive(isOngoing))
put("isClearable", JsonPrimitive(isClearable))
title?.let { put("title", JsonPrimitive(it)) }
text?.let { put("text", JsonPrimitive(it)) }
subText?.let { put("subText", JsonPrimitive(it)) }
category?.let { put("category", JsonPrimitive(it)) }
channelId?.let { put("channelId", JsonPrimitive(it)) }
}
}
data class DeviceNotificationSnapshot(
val enabled: Boolean,
val connected: Boolean,
val notifications: List<DeviceNotificationEntry>,
)
enum class NotificationActionKind {
Open,
Dismiss,
Reply,
}
data class NotificationActionRequest(
val key: String,
val kind: NotificationActionKind,
val replyText: String? = null,
)
data class NotificationActionResult(
val ok: Boolean,
val code: String? = null,
val message: String? = null,
)
internal fun actionRequiresClearableNotification(kind: NotificationActionKind): Boolean {
return kind == NotificationActionKind.Dismiss
}
private object DeviceNotificationStore {
private val lock = Any()
private var connected = false
private val byKey = LinkedHashMap<String, DeviceNotificationEntry>()
fun replace(entries: List<DeviceNotificationEntry>) {
synchronized(lock) {
byKey.clear()
for (entry in entries) {
byKey[entry.key] = entry
}
}
}
fun upsert(entry: DeviceNotificationEntry) {
synchronized(lock) {
byKey[entry.key] = entry
}
}
fun remove(key: String) {
synchronized(lock) {
byKey.remove(key)
}
}
fun setConnected(value: Boolean) {
synchronized(lock) {
connected = value
if (!value) {
byKey.clear()
}
}
}
fun snapshot(enabled: Boolean): DeviceNotificationSnapshot {
val (isConnected, entries) =
synchronized(lock) {
connected to byKey.values.sortedByDescending { it.postTimeMs }
}
return DeviceNotificationSnapshot(
enabled = enabled,
connected = isConnected,
notifications = entries,
)
}
}
class DeviceNotificationListenerService : NotificationListenerService() {
private val securePrefs by lazy { SecurePrefs(applicationContext) }
private val forwardingLimiter = NotificationBurstLimiter()
override fun onListenerConnected() {
super.onListenerConnected()
activeService = this
DeviceNotificationStore.setConnected(true)
refreshActiveNotifications()
}
override fun onListenerDisconnected() {
if (activeService === this) {
activeService = null
}
DeviceNotificationStore.setConnected(false)
super.onListenerDisconnected()
}
override fun onDestroy() {
if (activeService === this) {
activeService = null
}
super.onDestroy()
}
override fun onNotificationPosted(sbn: StatusBarNotification?) {
super.onNotificationPosted(sbn)
val entry = sbn?.toEntry() ?: return
DeviceNotificationStore.upsert(entry)
rememberRecentPackage(entry.packageName)
if (entry.packageName == packageName) {
return
}
val payload = notificationChangedPayload(entry) ?: return
emitNotificationsChanged(payload)
}
override fun onNotificationRemoved(sbn: StatusBarNotification?) {
super.onNotificationRemoved(sbn)
val removed = sbn ?: return
val key = removed.key.trim()
if (key.isEmpty()) {
return
}
DeviceNotificationStore.remove(key)
rememberRecentPackage(removed.packageName)
if (removed.packageName == packageName) {
return
}
val packageName = removed.packageName.trim()
val payload =
notificationChangedPayload(
entry = null,
change = "removed",
key = key,
packageName = packageName,
postTimeMs = removed.postTime,
isOngoing = removed.isOngoing,
isClearable = removed.isClearable,
) ?: return
emitNotificationsChanged(payload)
}
private fun notificationChangedPayload(entry: DeviceNotificationEntry): String? {
return notificationChangedPayload(
entry = entry,
change = "posted",
key = entry.key,
packageName = entry.packageName,
postTimeMs = entry.postTimeMs,
isOngoing = entry.isOngoing,
isClearable = entry.isClearable,
)
}
private fun notificationChangedPayload(
entry: DeviceNotificationEntry?,
change: String,
key: String,
packageName: String,
postTimeMs: Long,
isOngoing: Boolean,
isClearable: Boolean,
): String? {
val normalizedPackage = packageName.trim()
if (normalizedPackage.isEmpty()) {
return null
}
val policy = securePrefs.getNotificationForwardingPolicy(appPackageName = this.packageName)
if (!policy.enabled) {
return null
}
if (!policy.allowsPackage(normalizedPackage)) {
return null
}
val nowEpochMs = System.currentTimeMillis()
if (policy.isWithinQuietHours(nowEpochMs = nowEpochMs)) {
return null
}
if (!forwardingLimiter.allow(nowEpochMs, policy.maxEventsPerMinute)) {
return null
}
return buildJsonObject {
put("change", JsonPrimitive(change))
put("key", JsonPrimitive(key))
put("packageName", JsonPrimitive(normalizedPackage))
put("postTimeMs", JsonPrimitive(postTimeMs))
put("isOngoing", JsonPrimitive(isOngoing))
put("isClearable", JsonPrimitive(isClearable))
policy.sessionKey?.let { put("sessionKey", JsonPrimitive(it)) }
entry?.title?.let { put("title", JsonPrimitive(it)) }
entry?.text?.let { put("text", JsonPrimitive(it)) }
entry?.subText?.let { put("subText", JsonPrimitive(it)) }
entry?.category?.let { put("category", JsonPrimitive(it)) }
entry?.channelId?.let { put("channelId", JsonPrimitive(it)) }
}.toString()
}
private fun refreshActiveNotifications() {
val entries =
runCatching {
activeNotifications
?.mapNotNull { it.toEntry() }
?: emptyList()
}.getOrElse { emptyList() }
DeviceNotificationStore.replace(entries)
}
private fun StatusBarNotification.toEntry(): DeviceNotificationEntry {
val extras = notification.extras
val keyValue = key.takeIf { it.isNotBlank() } ?: "$packageName:$id:$postTime"
val title = sanitizeNotificationText(extras?.getCharSequence(Notification.EXTRA_TITLE))
val body =
sanitizeNotificationText(extras?.getCharSequence(Notification.EXTRA_BIG_TEXT))
?: sanitizeNotificationText(extras?.getCharSequence(Notification.EXTRA_TEXT))
val subText = sanitizeNotificationText(extras?.getCharSequence(Notification.EXTRA_SUB_TEXT))
return DeviceNotificationEntry(
key = keyValue,
packageName = packageName,
title = title,
text = body,
subText = subText,
category = notification.category?.trim()?.ifEmpty { null },
channelId = notification.channelId?.trim()?.ifEmpty { null },
postTimeMs = postTime,
isOngoing = isOngoing,
isClearable = isClearable,
)
}
companion object {
private const val recentPackagesPref = "notifications.forwarding.recentPackages"
private const val legacyRecentPackagesPref = "notifications.recentPackages"
private const val recentPackagesLimit = 64
@Volatile private var activeService: DeviceNotificationListenerService? = null
@Volatile private var nodeEventSink: ((event: String, payloadJson: String?) -> Unit)? = null
private fun serviceComponent(context: Context): ComponentName {
return ComponentName(context, DeviceNotificationListenerService::class.java)
}
fun setNodeEventSink(sink: ((event: String, payloadJson: String?) -> Unit)?) {
nodeEventSink = sink
}
private fun recentPackagesPrefs(context: Context) =
context.applicationContext.getSharedPreferences("openclaw.secure", Context.MODE_PRIVATE)
private fun migrateLegacyRecentPackagesIfNeeded(context: Context) {
val prefs = recentPackagesPrefs(context)
val hasNew = prefs.contains(recentPackagesPref)
val legacy = prefs.getString(legacyRecentPackagesPref, null)?.trim().orEmpty()
if (!hasNew && legacy.isNotEmpty()) {
prefs.edit().putString(recentPackagesPref, legacy).remove(legacyRecentPackagesPref).apply()
} else if (hasNew && prefs.contains(legacyRecentPackagesPref)) {
prefs.edit().remove(legacyRecentPackagesPref).apply()
}
}
fun recentPackages(context: Context): List<String> {
migrateLegacyRecentPackagesIfNeeded(context)
val prefs = recentPackagesPrefs(context)
val stored = prefs.getString(recentPackagesPref, null).orEmpty()
return stored
.split(',')
.map { it.trim() }
.filter { it.isNotEmpty() }
.distinct()
}
fun isAccessEnabled(context: Context): Boolean {
val manager = context.getSystemService(NotificationManager::class.java) ?: return false
return manager.isNotificationListenerAccessGranted(serviceComponent(context))
}
fun snapshot(context: Context, enabled: Boolean = isAccessEnabled(context)): DeviceNotificationSnapshot {
return DeviceNotificationStore.snapshot(enabled = enabled)
}
fun requestServiceRebind(context: Context) {
runCatching {
NotificationListenerService.requestRebind(serviceComponent(context))
}
}
fun executeAction(context: Context, request: NotificationActionRequest): NotificationActionResult {
if (!isAccessEnabled(context)) {
return NotificationActionResult(
ok = false,
code = "NOTIFICATIONS_DISABLED",
message = "NOTIFICATIONS_DISABLED: enable notification access in system Settings",
)
}
val service = activeService
?: return NotificationActionResult(
ok = false,
code = "NOTIFICATIONS_UNAVAILABLE",
message = "NOTIFICATIONS_UNAVAILABLE: notification listener not connected",
)
return service.executeActionInternal(request)
}
private fun emitNotificationsChanged(payloadJson: String) {
runCatching {
nodeEventSink?.invoke(NOTIFICATIONS_CHANGED_EVENT, payloadJson)
}
}
private fun rememberRecentPackage(packageName: String?) {
val service = activeService ?: return
val normalized = packageName?.trim().orEmpty()
if (normalized.isEmpty() || normalized == service.packageName) return
migrateLegacyRecentPackagesIfNeeded(service.applicationContext)
val prefs = recentPackagesPrefs(service.applicationContext)
val existing = prefs.getString(recentPackagesPref, null).orEmpty()
.split(',')
.map { it.trim() }
.filter { it.isNotEmpty() && it != normalized }
.take(recentPackagesLimit - 1)
val updated = listOf(normalized) + existing
prefs.edit().putString(recentPackagesPref, updated.joinToString(",")).apply()
}
}
private fun executeActionInternal(request: NotificationActionRequest): NotificationActionResult {
val sbn =
activeNotifications
?.firstOrNull { it.key == request.key }
?: return NotificationActionResult(
ok = false,
code = "NOTIFICATION_NOT_FOUND",
message = "NOTIFICATION_NOT_FOUND: notification key not found",
)
if (actionRequiresClearableNotification(request.kind) && !sbn.isClearable) {
return NotificationActionResult(
ok = false,
code = "NOTIFICATION_NOT_CLEARABLE",
message = "NOTIFICATION_NOT_CLEARABLE: notification is ongoing or protected",
)
}
return when (request.kind) {
NotificationActionKind.Open -> {
val pendingIntent = sbn.notification.contentIntent
?: return NotificationActionResult(
ok = false,
code = "ACTION_UNAVAILABLE",
message = "ACTION_UNAVAILABLE: notification has no open action",
)
runCatching {
pendingIntent.send()
}.fold(
onSuccess = { NotificationActionResult(ok = true) },
onFailure = { err ->
NotificationActionResult(
ok = false,
code = "ACTION_FAILED",
message = "ACTION_FAILED: ${err.message ?: "open failed"}",
)
},
)
}
NotificationActionKind.Dismiss -> {
runCatching {
cancelNotification(sbn.key)
DeviceNotificationStore.remove(sbn.key)
}.fold(
onSuccess = { NotificationActionResult(ok = true) },
onFailure = { err ->
NotificationActionResult(
ok = false,
code = "ACTION_FAILED",
message = "ACTION_FAILED: ${err.message ?: "dismiss failed"}",
)
},
)
}
NotificationActionKind.Reply -> {
val replyText = request.replyText?.trim().orEmpty()
if (replyText.isEmpty()) {
return NotificationActionResult(
ok = false,
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: replyText required for reply action",
)
}
val action =
sbn.notification.actions
?.firstOrNull { candidate ->
candidate.actionIntent != null && !candidate.remoteInputs.isNullOrEmpty()
}
?: return NotificationActionResult(
ok = false,
code = "ACTION_UNAVAILABLE",
message = "ACTION_UNAVAILABLE: notification has no reply action",
)
val remoteInputs = action.remoteInputs ?: emptyArray()
val fillInIntent = Intent()
val replyBundle = android.os.Bundle()
for (remoteInput in remoteInputs) {
replyBundle.putCharSequence(remoteInput.resultKey, replyText)
}
RemoteInput.addResultsToIntent(remoteInputs, fillInIntent, replyBundle)
runCatching {
action.actionIntent.send(this, 0, fillInIntent)
}.fold(
onSuccess = { NotificationActionResult(ok = true) },
onFailure = { err ->
NotificationActionResult(
ok = false,
code = "ACTION_FAILED",
message = "ACTION_FAILED: ${err.message ?: "reply failed"}",
)
},
)
}
}
}
}

View file

@ -0,0 +1,71 @@
package ai.openclaw.app.node
import ai.openclaw.app.SecurePrefs
import ai.openclaw.app.gateway.GatewaySession
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Job
import kotlinx.coroutines.delay
import kotlinx.coroutines.launch
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonArray
class GatewayEventHandler(
private val scope: CoroutineScope,
private val prefs: SecurePrefs,
private val json: Json,
private val operatorSession: GatewaySession,
private val isConnected: () -> Boolean,
) {
private var suppressWakeWordsSync = false
private var wakeWordsSyncJob: Job? = null
fun applyWakeWordsFromGateway(words: List<String>) {
suppressWakeWordsSync = true
prefs.setWakeWords(words)
suppressWakeWordsSync = false
}
fun scheduleWakeWordsSyncIfNeeded() {
if (suppressWakeWordsSync) return
if (!isConnected()) return
val snapshot = prefs.wakeWords.value
wakeWordsSyncJob?.cancel()
wakeWordsSyncJob =
scope.launch {
delay(650)
val jsonList = snapshot.joinToString(separator = ",") { it.toJsonString() }
val params = """{"triggers":[$jsonList]}"""
try {
operatorSession.request("voicewake.set", params)
} catch (_: Throwable) {
// ignore
}
}
}
suspend fun refreshWakeWordsFromGateway() {
if (!isConnected()) return
try {
val res = operatorSession.request("voicewake.get", "{}")
val payload = json.parseToJsonElement(res).asObjectOrNull() ?: return
val array = payload["triggers"] as? JsonArray ?: return
val triggers = array.mapNotNull { it.asStringOrNull() }
applyWakeWordsFromGateway(triggers)
} catch (_: Throwable) {
// ignore
}
}
fun handleVoiceWakeChangedEvent(payloadJson: String?) {
if (payloadJson.isNullOrBlank()) return
try {
val payload = json.parseToJsonElement(payloadJson).asObjectOrNull() ?: return
val array = payload["triggers"] as? JsonArray ?: return
val triggers = array.mapNotNull { it.asStringOrNull() }
applyWakeWordsFromGateway(triggers)
} catch (_: Throwable) {
// ignore
}
}
}

View file

@ -0,0 +1,258 @@
package ai.openclaw.app.node
import ai.openclaw.app.protocol.OpenClawCalendarCommand
import ai.openclaw.app.protocol.OpenClawCanvasA2UICommand
import ai.openclaw.app.protocol.OpenClawCanvasCommand
import ai.openclaw.app.protocol.OpenClawCameraCommand
import ai.openclaw.app.protocol.OpenClawCapability
import ai.openclaw.app.protocol.OpenClawCallLogCommand
import ai.openclaw.app.protocol.OpenClawContactsCommand
import ai.openclaw.app.protocol.OpenClawDeviceCommand
import ai.openclaw.app.protocol.OpenClawLocationCommand
import ai.openclaw.app.protocol.OpenClawMotionCommand
import ai.openclaw.app.protocol.OpenClawNotificationsCommand
import ai.openclaw.app.protocol.OpenClawPhotosCommand
import ai.openclaw.app.protocol.OpenClawSmsCommand
import ai.openclaw.app.protocol.OpenClawSystemCommand
data class NodeRuntimeFlags(
val cameraEnabled: Boolean,
val locationEnabled: Boolean,
val sendSmsAvailable: Boolean,
val readSmsAvailable: Boolean,
val smsSearchPossible: Boolean,
val callLogAvailable: Boolean,
val voiceWakeEnabled: Boolean,
val motionActivityAvailable: Boolean,
val motionPedometerAvailable: Boolean,
val debugBuild: Boolean,
)
enum class InvokeCommandAvailability {
Always,
CameraEnabled,
LocationEnabled,
SendSmsAvailable,
ReadSmsAvailable,
RequestableSmsSearchAvailable,
CallLogAvailable,
MotionActivityAvailable,
MotionPedometerAvailable,
DebugBuild,
}
enum class NodeCapabilityAvailability {
Always,
CameraEnabled,
LocationEnabled,
SmsAvailable,
CallLogAvailable,
VoiceWakeEnabled,
MotionAvailable,
}
data class NodeCapabilitySpec(
val name: String,
val availability: NodeCapabilityAvailability = NodeCapabilityAvailability.Always,
)
data class InvokeCommandSpec(
val name: String,
val requiresForeground: Boolean = false,
val availability: InvokeCommandAvailability = InvokeCommandAvailability.Always,
)
object InvokeCommandRegistry {
val capabilityManifest: List<NodeCapabilitySpec> =
listOf(
NodeCapabilitySpec(name = OpenClawCapability.Canvas.rawValue),
NodeCapabilitySpec(name = OpenClawCapability.Device.rawValue),
NodeCapabilitySpec(name = OpenClawCapability.Notifications.rawValue),
NodeCapabilitySpec(name = OpenClawCapability.System.rawValue),
NodeCapabilitySpec(
name = OpenClawCapability.Camera.rawValue,
availability = NodeCapabilityAvailability.CameraEnabled,
),
NodeCapabilitySpec(
name = OpenClawCapability.Sms.rawValue,
availability = NodeCapabilityAvailability.SmsAvailable,
),
NodeCapabilitySpec(
name = OpenClawCapability.VoiceWake.rawValue,
availability = NodeCapabilityAvailability.VoiceWakeEnabled,
),
NodeCapabilitySpec(
name = OpenClawCapability.Location.rawValue,
availability = NodeCapabilityAvailability.LocationEnabled,
),
NodeCapabilitySpec(name = OpenClawCapability.Photos.rawValue),
NodeCapabilitySpec(name = OpenClawCapability.Contacts.rawValue),
NodeCapabilitySpec(name = OpenClawCapability.Calendar.rawValue),
NodeCapabilitySpec(
name = OpenClawCapability.Motion.rawValue,
availability = NodeCapabilityAvailability.MotionAvailable,
),
NodeCapabilitySpec(
name = OpenClawCapability.CallLog.rawValue,
availability = NodeCapabilityAvailability.CallLogAvailable,
),
)
val all: List<InvokeCommandSpec> =
listOf(
InvokeCommandSpec(
name = OpenClawCanvasCommand.Present.rawValue,
requiresForeground = true,
),
InvokeCommandSpec(
name = OpenClawCanvasCommand.Hide.rawValue,
requiresForeground = true,
),
InvokeCommandSpec(
name = OpenClawCanvasCommand.Navigate.rawValue,
requiresForeground = true,
),
InvokeCommandSpec(
name = OpenClawCanvasCommand.Eval.rawValue,
requiresForeground = true,
),
InvokeCommandSpec(
name = OpenClawCanvasCommand.Snapshot.rawValue,
requiresForeground = true,
),
InvokeCommandSpec(
name = OpenClawCanvasA2UICommand.Push.rawValue,
requiresForeground = true,
),
InvokeCommandSpec(
name = OpenClawCanvasA2UICommand.PushJSONL.rawValue,
requiresForeground = true,
),
InvokeCommandSpec(
name = OpenClawCanvasA2UICommand.Reset.rawValue,
requiresForeground = true,
),
InvokeCommandSpec(
name = OpenClawSystemCommand.Notify.rawValue,
),
InvokeCommandSpec(
name = OpenClawCameraCommand.List.rawValue,
requiresForeground = true,
availability = InvokeCommandAvailability.CameraEnabled,
),
InvokeCommandSpec(
name = OpenClawCameraCommand.Snap.rawValue,
requiresForeground = true,
availability = InvokeCommandAvailability.CameraEnabled,
),
InvokeCommandSpec(
name = OpenClawCameraCommand.Clip.rawValue,
requiresForeground = true,
availability = InvokeCommandAvailability.CameraEnabled,
),
InvokeCommandSpec(
name = OpenClawLocationCommand.Get.rawValue,
availability = InvokeCommandAvailability.LocationEnabled,
),
InvokeCommandSpec(
name = OpenClawDeviceCommand.Status.rawValue,
),
InvokeCommandSpec(
name = OpenClawDeviceCommand.Info.rawValue,
),
InvokeCommandSpec(
name = OpenClawDeviceCommand.Permissions.rawValue,
),
InvokeCommandSpec(
name = OpenClawDeviceCommand.Health.rawValue,
),
InvokeCommandSpec(
name = OpenClawNotificationsCommand.List.rawValue,
),
InvokeCommandSpec(
name = OpenClawNotificationsCommand.Actions.rawValue,
),
InvokeCommandSpec(
name = OpenClawPhotosCommand.Latest.rawValue,
),
InvokeCommandSpec(
name = OpenClawContactsCommand.Search.rawValue,
),
InvokeCommandSpec(
name = OpenClawContactsCommand.Add.rawValue,
),
InvokeCommandSpec(
name = OpenClawCalendarCommand.Events.rawValue,
),
InvokeCommandSpec(
name = OpenClawCalendarCommand.Add.rawValue,
),
InvokeCommandSpec(
name = OpenClawMotionCommand.Activity.rawValue,
availability = InvokeCommandAvailability.MotionActivityAvailable,
),
InvokeCommandSpec(
name = OpenClawMotionCommand.Pedometer.rawValue,
availability = InvokeCommandAvailability.MotionPedometerAvailable,
),
InvokeCommandSpec(
name = OpenClawSmsCommand.Send.rawValue,
availability = InvokeCommandAvailability.SendSmsAvailable,
),
InvokeCommandSpec(
name = OpenClawSmsCommand.Search.rawValue,
availability = InvokeCommandAvailability.RequestableSmsSearchAvailable,
),
InvokeCommandSpec(
name = OpenClawCallLogCommand.Search.rawValue,
availability = InvokeCommandAvailability.CallLogAvailable,
),
InvokeCommandSpec(
name = "debug.logs",
availability = InvokeCommandAvailability.DebugBuild,
),
InvokeCommandSpec(
name = "debug.ed25519",
availability = InvokeCommandAvailability.DebugBuild,
),
)
private val byNameInternal: Map<String, InvokeCommandSpec> = all.associateBy { it.name }
fun find(command: String): InvokeCommandSpec? = byNameInternal[command]
fun advertisedCapabilities(flags: NodeRuntimeFlags): List<String> {
return capabilityManifest
.filter { spec ->
when (spec.availability) {
NodeCapabilityAvailability.Always -> true
NodeCapabilityAvailability.CameraEnabled -> flags.cameraEnabled
NodeCapabilityAvailability.LocationEnabled -> flags.locationEnabled
NodeCapabilityAvailability.SmsAvailable -> flags.sendSmsAvailable || flags.readSmsAvailable
NodeCapabilityAvailability.CallLogAvailable -> flags.callLogAvailable
NodeCapabilityAvailability.VoiceWakeEnabled -> flags.voiceWakeEnabled
NodeCapabilityAvailability.MotionAvailable -> flags.motionActivityAvailable || flags.motionPedometerAvailable
}
}
.map { it.name }
}
fun advertisedCommands(flags: NodeRuntimeFlags): List<String> {
return all
.filter { spec ->
when (spec.availability) {
InvokeCommandAvailability.Always -> true
InvokeCommandAvailability.CameraEnabled -> flags.cameraEnabled
InvokeCommandAvailability.LocationEnabled -> flags.locationEnabled
InvokeCommandAvailability.SendSmsAvailable -> flags.sendSmsAvailable
InvokeCommandAvailability.ReadSmsAvailable -> flags.readSmsAvailable
InvokeCommandAvailability.RequestableSmsSearchAvailable -> flags.smsSearchPossible
InvokeCommandAvailability.CallLogAvailable -> flags.callLogAvailable
InvokeCommandAvailability.MotionActivityAvailable -> flags.motionActivityAvailable
InvokeCommandAvailability.MotionPedometerAvailable -> flags.motionPedometerAvailable
InvokeCommandAvailability.DebugBuild -> flags.debugBuild
}
}
.map { it.name }
}
}

View file

@ -0,0 +1,338 @@
package ai.openclaw.app.node
import ai.openclaw.app.gateway.GatewaySession
import ai.openclaw.app.protocol.OpenClawCalendarCommand
import ai.openclaw.app.protocol.OpenClawCanvasA2UICommand
import ai.openclaw.app.protocol.OpenClawCanvasCommand
import ai.openclaw.app.protocol.OpenClawCameraCommand
import ai.openclaw.app.protocol.OpenClawCallLogCommand
import ai.openclaw.app.protocol.OpenClawContactsCommand
import ai.openclaw.app.protocol.OpenClawDeviceCommand
import ai.openclaw.app.protocol.OpenClawLocationCommand
import ai.openclaw.app.protocol.OpenClawMotionCommand
import ai.openclaw.app.protocol.OpenClawNotificationsCommand
import ai.openclaw.app.protocol.OpenClawSmsCommand
import ai.openclaw.app.protocol.OpenClawSystemCommand
internal enum class SmsSearchAvailabilityReason {
Available,
PermissionRequired,
Unavailable,
}
internal fun classifySmsSearchAvailability(
readSmsAvailable: Boolean,
smsFeatureEnabled: Boolean,
smsTelephonyAvailable: Boolean,
): SmsSearchAvailabilityReason {
if (readSmsAvailable) return SmsSearchAvailabilityReason.Available
if (!smsFeatureEnabled || !smsTelephonyAvailable) return SmsSearchAvailabilityReason.Unavailable
return SmsSearchAvailabilityReason.PermissionRequired
}
internal fun smsSearchAvailabilityError(
readSmsAvailable: Boolean,
smsFeatureEnabled: Boolean,
smsTelephonyAvailable: Boolean,
): GatewaySession.InvokeResult? {
return when (
classifySmsSearchAvailability(
readSmsAvailable = readSmsAvailable,
smsFeatureEnabled = smsFeatureEnabled,
smsTelephonyAvailable = smsTelephonyAvailable,
)
) {
SmsSearchAvailabilityReason.Available,
SmsSearchAvailabilityReason.PermissionRequired -> null
SmsSearchAvailabilityReason.Unavailable ->
GatewaySession.InvokeResult.error(
code = "SMS_UNAVAILABLE",
message = "SMS_UNAVAILABLE: SMS not available on this device",
)
}
}
class InvokeDispatcher(
private val canvas: CanvasController,
private val cameraHandler: CameraHandler,
private val locationHandler: LocationHandler,
private val deviceHandler: DeviceHandler,
private val notificationsHandler: NotificationsHandler,
private val systemHandler: SystemHandler,
private val photosHandler: PhotosHandler,
private val contactsHandler: ContactsHandler,
private val calendarHandler: CalendarHandler,
private val motionHandler: MotionHandler,
private val smsHandler: SmsHandler,
private val a2uiHandler: A2UIHandler,
private val debugHandler: DebugHandler,
private val callLogHandler: CallLogHandler,
private val isForeground: () -> Boolean,
private val cameraEnabled: () -> Boolean,
private val locationEnabled: () -> Boolean,
private val sendSmsAvailable: () -> Boolean,
private val readSmsAvailable: () -> Boolean,
private val smsFeatureEnabled: () -> Boolean,
private val smsTelephonyAvailable: () -> Boolean,
private val callLogAvailable: () -> Boolean,
private val debugBuild: () -> Boolean,
private val refreshNodeCanvasCapability: suspend () -> Boolean,
private val onCanvasA2uiPush: () -> Unit,
private val onCanvasA2uiReset: () -> Unit,
private val motionActivityAvailable: () -> Boolean,
private val motionPedometerAvailable: () -> Boolean,
) {
suspend fun handleInvoke(command: String, paramsJson: String?): GatewaySession.InvokeResult {
val spec =
InvokeCommandRegistry.find(command)
?: return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: unknown command",
)
if (spec.requiresForeground && !isForeground()) {
return GatewaySession.InvokeResult.error(
code = "NODE_BACKGROUND_UNAVAILABLE",
message = "NODE_BACKGROUND_UNAVAILABLE: canvas/camera/screen commands require foreground",
)
}
availabilityError(spec.availability)?.let { return it }
return when (command) {
// Canvas commands
OpenClawCanvasCommand.Present.rawValue -> {
val url = CanvasController.parseNavigateUrl(paramsJson)
canvas.navigate(url)
GatewaySession.InvokeResult.ok(null)
}
OpenClawCanvasCommand.Hide.rawValue -> GatewaySession.InvokeResult.ok(null)
OpenClawCanvasCommand.Navigate.rawValue -> {
val url = CanvasController.parseNavigateUrl(paramsJson)
canvas.navigate(url)
GatewaySession.InvokeResult.ok(null)
}
OpenClawCanvasCommand.Eval.rawValue -> {
val js =
CanvasController.parseEvalJs(paramsJson)
?: return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: javaScript required",
)
withCanvasAvailable {
val result = canvas.eval(js)
GatewaySession.InvokeResult.ok("""{"result":${result.toJsonString()}}""")
}
}
OpenClawCanvasCommand.Snapshot.rawValue -> {
val snapshotParams = CanvasController.parseSnapshotParams(paramsJson)
withCanvasAvailable {
val base64 =
canvas.snapshotBase64(
format = snapshotParams.format,
quality = snapshotParams.quality,
maxWidth = snapshotParams.maxWidth,
)
GatewaySession.InvokeResult.ok("""{"format":"${snapshotParams.format.rawValue}","base64":"$base64"}""")
}
}
// A2UI commands
OpenClawCanvasA2UICommand.Reset.rawValue ->
withReadyA2ui {
withCanvasAvailable {
val res = canvas.eval(A2UIHandler.a2uiResetJS)
onCanvasA2uiReset()
GatewaySession.InvokeResult.ok(res)
}
}
OpenClawCanvasA2UICommand.Push.rawValue, OpenClawCanvasA2UICommand.PushJSONL.rawValue -> {
val messages =
try {
a2uiHandler.decodeA2uiMessages(command, paramsJson)
} catch (err: Throwable) {
return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = err.message ?: "invalid A2UI payload"
)
}
withReadyA2ui {
withCanvasAvailable {
val js = A2UIHandler.a2uiApplyMessagesJS(messages)
val res = canvas.eval(js)
onCanvasA2uiPush()
GatewaySession.InvokeResult.ok(res)
}
}
}
// Camera commands
OpenClawCameraCommand.List.rawValue -> cameraHandler.handleList(paramsJson)
OpenClawCameraCommand.Snap.rawValue -> cameraHandler.handleSnap(paramsJson)
OpenClawCameraCommand.Clip.rawValue -> cameraHandler.handleClip(paramsJson)
// Location command
OpenClawLocationCommand.Get.rawValue -> locationHandler.handleLocationGet(paramsJson)
// Device commands
OpenClawDeviceCommand.Status.rawValue -> deviceHandler.handleDeviceStatus(paramsJson)
OpenClawDeviceCommand.Info.rawValue -> deviceHandler.handleDeviceInfo(paramsJson)
OpenClawDeviceCommand.Permissions.rawValue -> deviceHandler.handleDevicePermissions(paramsJson)
OpenClawDeviceCommand.Health.rawValue -> deviceHandler.handleDeviceHealth(paramsJson)
// Notifications command
OpenClawNotificationsCommand.List.rawValue -> notificationsHandler.handleNotificationsList(paramsJson)
OpenClawNotificationsCommand.Actions.rawValue -> notificationsHandler.handleNotificationsActions(paramsJson)
// System command
OpenClawSystemCommand.Notify.rawValue -> systemHandler.handleSystemNotify(paramsJson)
// Photos command
ai.openclaw.app.protocol.OpenClawPhotosCommand.Latest.rawValue -> photosHandler.handlePhotosLatest(
paramsJson,
)
// Contacts command
OpenClawContactsCommand.Search.rawValue -> contactsHandler.handleContactsSearch(paramsJson)
OpenClawContactsCommand.Add.rawValue -> contactsHandler.handleContactsAdd(paramsJson)
// Calendar command
OpenClawCalendarCommand.Events.rawValue -> calendarHandler.handleCalendarEvents(paramsJson)
OpenClawCalendarCommand.Add.rawValue -> calendarHandler.handleCalendarAdd(paramsJson)
// Motion command
OpenClawMotionCommand.Activity.rawValue -> motionHandler.handleMotionActivity(paramsJson)
OpenClawMotionCommand.Pedometer.rawValue -> motionHandler.handleMotionPedometer(paramsJson)
// SMS command
OpenClawSmsCommand.Send.rawValue -> smsHandler.handleSmsSend(paramsJson)
OpenClawSmsCommand.Search.rawValue -> smsHandler.handleSmsSearch(paramsJson)
// CallLog command
OpenClawCallLogCommand.Search.rawValue -> callLogHandler.handleCallLogSearch(paramsJson)
// Debug commands
"debug.ed25519" -> debugHandler.handleEd25519()
"debug.logs" -> debugHandler.handleLogs()
else -> GatewaySession.InvokeResult.error(code = "INVALID_REQUEST", message = "INVALID_REQUEST: unknown command")
}
}
private suspend fun withReadyA2ui(
block: suspend () -> GatewaySession.InvokeResult,
): GatewaySession.InvokeResult {
var a2uiUrl = a2uiHandler.resolveA2uiHostUrl()
?: return GatewaySession.InvokeResult.error(
code = "A2UI_HOST_NOT_CONFIGURED",
message = "A2UI_HOST_NOT_CONFIGURED: gateway did not advertise canvas host",
)
val readyOnFirstCheck = a2uiHandler.ensureA2uiReady(a2uiUrl)
if (!readyOnFirstCheck) {
if (!refreshNodeCanvasCapability()) {
return GatewaySession.InvokeResult.error(
code = "A2UI_HOST_UNAVAILABLE",
message = "A2UI_HOST_UNAVAILABLE: A2UI host not reachable",
)
}
a2uiUrl = a2uiHandler.resolveA2uiHostUrl()
?: return GatewaySession.InvokeResult.error(
code = "A2UI_HOST_NOT_CONFIGURED",
message = "A2UI_HOST_NOT_CONFIGURED: gateway did not advertise canvas host",
)
if (!a2uiHandler.ensureA2uiReady(a2uiUrl)) {
return GatewaySession.InvokeResult.error(
code = "A2UI_HOST_UNAVAILABLE",
message = "A2UI_HOST_UNAVAILABLE: A2UI host not reachable",
)
}
}
return block()
}
private suspend fun withCanvasAvailable(
block: suspend () -> GatewaySession.InvokeResult,
): GatewaySession.InvokeResult {
return try {
block()
} catch (_: Throwable) {
GatewaySession.InvokeResult.error(
code = "NODE_BACKGROUND_UNAVAILABLE",
message = "NODE_BACKGROUND_UNAVAILABLE: canvas unavailable",
)
}
}
private fun availabilityError(availability: InvokeCommandAvailability): GatewaySession.InvokeResult? {
return when (availability) {
InvokeCommandAvailability.Always -> null
InvokeCommandAvailability.CameraEnabled ->
if (cameraEnabled()) {
null
} else {
GatewaySession.InvokeResult.error(
code = "CAMERA_DISABLED",
message = "CAMERA_DISABLED: enable Camera in Settings",
)
}
InvokeCommandAvailability.LocationEnabled ->
if (locationEnabled()) {
null
} else {
GatewaySession.InvokeResult.error(
code = "LOCATION_DISABLED",
message = "LOCATION_DISABLED: enable Location in Settings",
)
}
InvokeCommandAvailability.MotionActivityAvailable ->
if (motionActivityAvailable()) {
null
} else {
GatewaySession.InvokeResult.error(
code = "MOTION_UNAVAILABLE",
message = "MOTION_UNAVAILABLE: accelerometer not available",
)
}
InvokeCommandAvailability.MotionPedometerAvailable ->
if (motionPedometerAvailable()) {
null
} else {
GatewaySession.InvokeResult.error(
code = "PEDOMETER_UNAVAILABLE",
message = "PEDOMETER_UNAVAILABLE: step counter not available",
)
}
InvokeCommandAvailability.SendSmsAvailable ->
if (sendSmsAvailable()) {
null
} else {
GatewaySession.InvokeResult.error(
code = "SMS_UNAVAILABLE",
message = "SMS_UNAVAILABLE: SMS not available on this device",
)
}
InvokeCommandAvailability.ReadSmsAvailable,
InvokeCommandAvailability.RequestableSmsSearchAvailable ->
smsSearchAvailabilityError(
readSmsAvailable = readSmsAvailable(),
smsFeatureEnabled = smsFeatureEnabled(),
smsTelephonyAvailable = smsTelephonyAvailable(),
)
InvokeCommandAvailability.CallLogAvailable ->
if (callLogAvailable()) {
null
} else {
GatewaySession.InvokeResult.error(
code = "CALL_LOG_UNAVAILABLE",
message = "CALL_LOG_UNAVAILABLE: call log not available on this build",
)
}
InvokeCommandAvailability.DebugBuild ->
if (debugBuild()) {
null
} else {
GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: unknown command",
)
}
}
}
}

View file

@ -0,0 +1,61 @@
package ai.openclaw.app.node
import kotlin.math.max
import kotlin.math.min
import kotlin.math.roundToInt
internal data class JpegSizeLimiterResult(
val bytes: ByteArray,
val width: Int,
val height: Int,
val quality: Int,
)
internal object JpegSizeLimiter {
fun compressToLimit(
initialWidth: Int,
initialHeight: Int,
startQuality: Int,
maxBytes: Int,
minQuality: Int = 20,
minSize: Int = 256,
scaleStep: Double = 0.85,
maxScaleAttempts: Int = 6,
maxQualityAttempts: Int = 6,
encode: (width: Int, height: Int, quality: Int) -> ByteArray,
): JpegSizeLimiterResult {
require(initialWidth > 0 && initialHeight > 0) { "Invalid image size" }
require(maxBytes > 0) { "Invalid maxBytes" }
var width = initialWidth
var height = initialHeight
val clampedStartQuality = startQuality.coerceIn(minQuality, 100)
var best = JpegSizeLimiterResult(bytes = encode(width, height, clampedStartQuality), width = width, height = height, quality = clampedStartQuality)
if (best.bytes.size <= maxBytes) return best
repeat(maxScaleAttempts) {
var quality = clampedStartQuality
repeat(maxQualityAttempts) {
val bytes = encode(width, height, quality)
best = JpegSizeLimiterResult(bytes = bytes, width = width, height = height, quality = quality)
if (bytes.size <= maxBytes) return best
if (quality <= minQuality) return@repeat
quality = max(minQuality, (quality * 0.75).roundToInt())
}
val minScale = (minSize.toDouble() / min(width, height).toDouble()).coerceAtMost(1.0)
val nextScale = max(scaleStep, minScale)
val nextWidth = max(minSize, (width * nextScale).roundToInt())
val nextHeight = max(minSize, (height * nextScale).roundToInt())
if (nextWidth == width && nextHeight == height) return@repeat
width = min(nextWidth, width)
height = min(nextHeight, height)
}
if (best.bytes.size > maxBytes) {
throw IllegalStateException("CAMERA_TOO_LARGE: ${best.bytes.size} bytes > $maxBytes bytes")
}
return best
}
}

View file

@ -0,0 +1,112 @@
package ai.openclaw.app.node
import android.Manifest
import android.content.Context
import android.content.pm.PackageManager
import android.location.Location
import android.location.LocationManager
import android.os.CancellationSignal
import androidx.core.content.ContextCompat
import java.time.Instant
import java.time.format.DateTimeFormatter
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import kotlinx.coroutines.withTimeout
import kotlinx.coroutines.suspendCancellableCoroutine
class LocationCaptureManager(private val context: Context) {
data class Payload(val payloadJson: String)
suspend fun getLocation(
desiredProviders: List<String>,
maxAgeMs: Long?,
timeoutMs: Long,
isPrecise: Boolean,
): Payload =
withContext(Dispatchers.Main) {
val manager = context.getSystemService(Context.LOCATION_SERVICE) as LocationManager
if (!manager.isProviderEnabled(LocationManager.GPS_PROVIDER) &&
!manager.isProviderEnabled(LocationManager.NETWORK_PROVIDER)
) {
throw IllegalStateException("LOCATION_UNAVAILABLE: no location providers enabled")
}
val cached = bestLastKnown(manager, desiredProviders, maxAgeMs)
val location =
cached ?: requestCurrent(manager, desiredProviders, timeoutMs)
val timestamp = DateTimeFormatter.ISO_INSTANT.format(Instant.ofEpochMilli(location.time))
val source = location.provider
val altitudeMeters = if (location.hasAltitude()) location.altitude else null
val speedMps = if (location.hasSpeed()) location.speed.toDouble() else null
val headingDeg = if (location.hasBearing()) location.bearing.toDouble() else null
Payload(
buildString {
append("{\"lat\":")
append(location.latitude)
append(",\"lon\":")
append(location.longitude)
append(",\"accuracyMeters\":")
append(location.accuracy.toDouble())
if (altitudeMeters != null) append(",\"altitudeMeters\":").append(altitudeMeters)
if (speedMps != null) append(",\"speedMps\":").append(speedMps)
if (headingDeg != null) append(",\"headingDeg\":").append(headingDeg)
append(",\"timestamp\":\"").append(timestamp).append('"')
append(",\"isPrecise\":").append(isPrecise)
append(",\"source\":\"").append(source).append('"')
append('}')
},
)
}
private fun bestLastKnown(
manager: LocationManager,
providers: List<String>,
maxAgeMs: Long?,
): Location? {
val fineOk =
ContextCompat.checkSelfPermission(context, Manifest.permission.ACCESS_FINE_LOCATION) ==
PackageManager.PERMISSION_GRANTED
val coarseOk =
ContextCompat.checkSelfPermission(context, Manifest.permission.ACCESS_COARSE_LOCATION) ==
PackageManager.PERMISSION_GRANTED
if (!fineOk && !coarseOk) {
throw IllegalStateException("LOCATION_PERMISSION_REQUIRED: grant Location permission")
}
val now = System.currentTimeMillis()
val candidates =
providers.mapNotNull { provider -> manager.getLastKnownLocation(provider) }
val freshest = candidates.maxByOrNull { it.time } ?: return null
if (maxAgeMs != null && now - freshest.time > maxAgeMs) return null
return freshest
}
private suspend fun requestCurrent(
manager: LocationManager,
providers: List<String>,
timeoutMs: Long,
): Location {
val fineOk =
ContextCompat.checkSelfPermission(context, Manifest.permission.ACCESS_FINE_LOCATION) ==
PackageManager.PERMISSION_GRANTED
val coarseOk =
ContextCompat.checkSelfPermission(context, Manifest.permission.ACCESS_COARSE_LOCATION) ==
PackageManager.PERMISSION_GRANTED
if (!fineOk && !coarseOk) {
throw IllegalStateException("LOCATION_PERMISSION_REQUIRED: grant Location permission")
}
val resolved =
providers.firstOrNull { manager.isProviderEnabled(it) }
?: throw IllegalStateException("LOCATION_UNAVAILABLE: no providers available")
val location = withTimeout(timeoutMs.coerceAtLeast(1)) {
suspendCancellableCoroutine<Location?> { cont ->
val signal = CancellationSignal()
cont.invokeOnCancellation { signal.cancel() }
manager.getCurrentLocation(resolved, signal, context.mainExecutor) { location ->
cont.resume(location) { _, _, _ -> }
}
}
}
return location ?: throw IllegalStateException("LOCATION_UNAVAILABLE: no fix")
}
}

View file

@ -0,0 +1,158 @@
package ai.openclaw.app.node
import android.Manifest
import android.content.Context
import android.content.pm.PackageManager
import android.location.LocationManager
import androidx.core.content.ContextCompat
import ai.openclaw.app.gateway.GatewaySession
import kotlinx.coroutines.TimeoutCancellationException
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonPrimitive
internal interface LocationDataSource {
fun hasFinePermission(context: Context): Boolean
fun hasCoarsePermission(context: Context): Boolean
suspend fun fetchLocation(
desiredProviders: List<String>,
maxAgeMs: Long?,
timeoutMs: Long,
isPrecise: Boolean,
): LocationCaptureManager.Payload
}
private class DefaultLocationDataSource(
private val capture: LocationCaptureManager,
) : LocationDataSource {
override fun hasFinePermission(context: Context): Boolean =
ContextCompat.checkSelfPermission(context, Manifest.permission.ACCESS_FINE_LOCATION) ==
PackageManager.PERMISSION_GRANTED
override fun hasCoarsePermission(context: Context): Boolean =
ContextCompat.checkSelfPermission(context, Manifest.permission.ACCESS_COARSE_LOCATION) ==
PackageManager.PERMISSION_GRANTED
override suspend fun fetchLocation(
desiredProviders: List<String>,
maxAgeMs: Long?,
timeoutMs: Long,
isPrecise: Boolean,
): LocationCaptureManager.Payload =
capture.getLocation(
desiredProviders = desiredProviders,
maxAgeMs = maxAgeMs,
timeoutMs = timeoutMs,
isPrecise = isPrecise,
)
}
class LocationHandler private constructor(
private val appContext: Context,
private val dataSource: LocationDataSource,
private val json: Json,
private val isForeground: () -> Boolean,
private val locationPreciseEnabled: () -> Boolean,
) {
constructor(
appContext: Context,
location: LocationCaptureManager,
json: Json,
isForeground: () -> Boolean,
locationPreciseEnabled: () -> Boolean,
) : this(
appContext = appContext,
dataSource = DefaultLocationDataSource(location),
json = json,
isForeground = isForeground,
locationPreciseEnabled = locationPreciseEnabled,
)
fun hasFineLocationPermission(): Boolean = dataSource.hasFinePermission(appContext)
fun hasCoarseLocationPermission(): Boolean = dataSource.hasCoarsePermission(appContext)
companion object {
internal fun forTesting(
appContext: Context,
dataSource: LocationDataSource,
json: Json = Json { ignoreUnknownKeys = true },
isForeground: () -> Boolean = { true },
locationPreciseEnabled: () -> Boolean = { true },
): LocationHandler =
LocationHandler(
appContext = appContext,
dataSource = dataSource,
json = json,
isForeground = isForeground,
locationPreciseEnabled = locationPreciseEnabled,
)
}
suspend fun handleLocationGet(paramsJson: String?): GatewaySession.InvokeResult {
if (!isForeground()) {
return GatewaySession.InvokeResult.error(
code = "LOCATION_BACKGROUND_UNAVAILABLE",
message = "LOCATION_BACKGROUND_UNAVAILABLE: location requires OpenClaw to stay open",
)
}
if (!dataSource.hasFinePermission(appContext) && !dataSource.hasCoarsePermission(appContext)) {
return GatewaySession.InvokeResult.error(
code = "LOCATION_PERMISSION_REQUIRED",
message = "LOCATION_PERMISSION_REQUIRED: grant Location permission",
)
}
val (maxAgeMs, timeoutMs, desiredAccuracy) = parseLocationParams(paramsJson)
val preciseEnabled = locationPreciseEnabled()
val accuracy =
when (desiredAccuracy) {
"precise" -> if (preciseEnabled && dataSource.hasFinePermission(appContext)) "precise" else "balanced"
"coarse" -> "coarse"
else -> if (preciseEnabled && dataSource.hasFinePermission(appContext)) "precise" else "balanced"
}
val providers =
when (accuracy) {
"precise" -> listOf(LocationManager.GPS_PROVIDER, LocationManager.NETWORK_PROVIDER)
"coarse" -> listOf(LocationManager.NETWORK_PROVIDER, LocationManager.GPS_PROVIDER)
else -> listOf(LocationManager.NETWORK_PROVIDER, LocationManager.GPS_PROVIDER)
}
try {
val payload =
dataSource.fetchLocation(
desiredProviders = providers,
maxAgeMs = maxAgeMs,
timeoutMs = timeoutMs,
isPrecise = accuracy == "precise",
)
return GatewaySession.InvokeResult.ok(payload.payloadJson)
} catch (err: TimeoutCancellationException) {
return GatewaySession.InvokeResult.error(
code = "LOCATION_TIMEOUT",
message = "LOCATION_TIMEOUT: no fix in time",
)
} catch (err: Throwable) {
val message = err.message ?: "LOCATION_UNAVAILABLE: no fix"
return GatewaySession.InvokeResult.error(code = "LOCATION_UNAVAILABLE", message = message)
}
}
private fun parseLocationParams(paramsJson: String?): Triple<Long?, Long, String?> {
if (paramsJson.isNullOrBlank()) {
return Triple(null, 10_000L, null)
}
val root =
try {
json.parseToJsonElement(paramsJson).asObjectOrNull()
} catch (_: Throwable) {
null
}
val maxAgeMs = (root?.get("maxAgeMs") as? JsonPrimitive)?.content?.toLongOrNull()
val timeoutMs =
(root?.get("timeoutMs") as? JsonPrimitive)?.content?.toLongOrNull()?.coerceIn(1_000L, 60_000L)
?: 10_000L
val desiredAccuracy =
(root?.get("desiredAccuracy") as? JsonPrimitive)?.content?.trim()?.lowercase()
return Triple(maxAgeMs, timeoutMs, desiredAccuracy)
}
}

View file

@ -0,0 +1,371 @@
package ai.openclaw.app.node
import android.Manifest
import android.content.Context
import android.hardware.Sensor
import android.hardware.SensorEvent
import android.hardware.SensorEventListener
import android.hardware.SensorManager
import android.os.SystemClock
import androidx.core.content.ContextCompat
import ai.openclaw.app.gateway.GatewaySession
import java.time.Instant
import kotlinx.coroutines.InternalCoroutinesApi
import kotlinx.coroutines.suspendCancellableCoroutine
import kotlinx.coroutines.withTimeoutOrNull
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.buildJsonArray
import kotlinx.serialization.json.buildJsonObject
import kotlinx.serialization.json.put
import kotlin.math.abs
import kotlin.math.max
import kotlin.math.sqrt
private const val ACCELEROMETER_SAMPLE_TARGET = 20
private const val ACCELEROMETER_SAMPLE_TIMEOUT_MS = 6_000L
internal data class MotionActivityRequest(
val startISO: String?,
val endISO: String?,
val limit: Int,
)
internal data class MotionPedometerRequest(
val startISO: String?,
val endISO: String?,
)
internal data class MotionActivityRecord(
val startISO: String,
val endISO: String,
val confidence: String,
val isWalking: Boolean,
val isRunning: Boolean,
val isCycling: Boolean,
val isAutomotive: Boolean,
val isStationary: Boolean,
val isUnknown: Boolean,
)
internal data class PedometerRecord(
val startISO: String,
val endISO: String,
val steps: Int?,
val distanceMeters: Double?,
val floorsAscended: Int?,
val floorsDescended: Int?,
)
internal interface MotionDataSource {
fun isActivityAvailable(context: Context): Boolean
fun isPedometerAvailable(context: Context): Boolean
fun isAvailable(context: Context): Boolean = isActivityAvailable(context) || isPedometerAvailable(context)
fun hasPermission(context: Context): Boolean
suspend fun activity(context: Context, request: MotionActivityRequest): MotionActivityRecord
suspend fun pedometer(context: Context, request: MotionPedometerRequest): PedometerRecord
}
private object SystemMotionDataSource : MotionDataSource {
override fun isActivityAvailable(context: Context): Boolean {
val sensorManager = context.getSystemService(SensorManager::class.java)
return sensorManager?.getDefaultSensor(Sensor.TYPE_ACCELEROMETER) != null
}
override fun isPedometerAvailable(context: Context): Boolean {
val sensorManager = context.getSystemService(SensorManager::class.java)
return sensorManager?.getDefaultSensor(Sensor.TYPE_STEP_COUNTER) != null
}
override fun hasPermission(context: Context): Boolean {
return ContextCompat.checkSelfPermission(context, Manifest.permission.ACTIVITY_RECOGNITION) ==
android.content.pm.PackageManager.PERMISSION_GRANTED
}
override suspend fun activity(context: Context, request: MotionActivityRequest): MotionActivityRecord {
if (!request.startISO.isNullOrBlank() || !request.endISO.isNullOrBlank()) {
throw IllegalArgumentException("MOTION_RANGE_UNAVAILABLE: historical activity range not supported on Android")
}
val sensorManager = context.getSystemService(SensorManager::class.java)
?: throw IllegalStateException("MOTION_UNAVAILABLE: sensor manager unavailable")
val accelerometer = sensorManager.getDefaultSensor(Sensor.TYPE_ACCELEROMETER)
?: throw IllegalStateException("MOTION_UNAVAILABLE: accelerometer not available")
val sample = readAccelerometerSample(sensorManager, accelerometer)
?: throw IllegalStateException("MOTION_UNAVAILABLE: no accelerometer sample")
val end = Instant.now()
val start = end.minusSeconds(2)
val classification = classifyActivity(sample.averageDelta)
return MotionActivityRecord(
startISO = start.toString(),
endISO = end.toString(),
confidence = classifyConfidence(sample.samples, sample.averageDelta),
isWalking = classification == "walking",
isRunning = classification == "running",
isCycling = false,
isAutomotive = false,
isStationary = classification == "stationary",
isUnknown = classification == "unknown",
)
}
override suspend fun pedometer(context: Context, request: MotionPedometerRequest): PedometerRecord {
if (!request.startISO.isNullOrBlank() || !request.endISO.isNullOrBlank()) {
throw IllegalArgumentException("PEDOMETER_RANGE_UNAVAILABLE: historical pedometer range not supported on Android")
}
val sensorManager = context.getSystemService(SensorManager::class.java)
?: throw IllegalStateException("PEDOMETER_UNAVAILABLE: sensor manager unavailable")
val stepCounter = sensorManager.getDefaultSensor(Sensor.TYPE_STEP_COUNTER)
?: throw IllegalStateException("PEDOMETER_UNAVAILABLE: step counting not supported")
val steps = readStepCounter(sensorManager, stepCounter)
?: throw IllegalStateException("PEDOMETER_UNAVAILABLE: no step counter sample")
val bootMs = System.currentTimeMillis() - SystemClock.elapsedRealtime()
return PedometerRecord(
startISO = Instant.ofEpochMilli(max(0L, bootMs)).toString(),
endISO = Instant.now().toString(),
steps = steps,
distanceMeters = null,
floorsAscended = null,
floorsDescended = null,
)
}
private data class AccelerometerSample(
val samples: Int,
val averageDelta: Double,
)
@OptIn(InternalCoroutinesApi::class)
private suspend fun readStepCounter(sensorManager: SensorManager, sensor: Sensor): Int? {
val sample =
withTimeoutOrNull(1200L) {
suspendCancellableCoroutine<Float?> { cont ->
val listener =
object : SensorEventListener {
override fun onSensorChanged(event: SensorEvent?) {
val value = event?.values?.firstOrNull()
val token = cont.tryResume(value) ?: return
cont.completeResume(token)
sensorManager.unregisterListener(this)
}
override fun onAccuracyChanged(sensor: Sensor?, accuracy: Int) = Unit
}
val registered = sensorManager.registerListener(listener, sensor, SensorManager.SENSOR_DELAY_NORMAL)
if (!registered) {
sensorManager.unregisterListener(listener)
cont.resume(null) { _, _, _ -> }
return@suspendCancellableCoroutine
}
cont.invokeOnCancellation { sensorManager.unregisterListener(listener) }
}
}
return sample?.toInt()?.takeIf { it >= 0 }
}
@OptIn(InternalCoroutinesApi::class)
private suspend fun readAccelerometerSample(
sensorManager: SensorManager,
sensor: Sensor,
): AccelerometerSample? {
val sample =
withTimeoutOrNull(ACCELEROMETER_SAMPLE_TIMEOUT_MS) {
suspendCancellableCoroutine<AccelerometerSample?> { cont ->
var count = 0
var sumDelta = 0.0
val listener =
object : SensorEventListener {
override fun onSensorChanged(event: SensorEvent?) {
val values = event?.values ?: return
if (values.size < 3) return
val magnitude =
sqrt(
values[0] * values[0] +
values[1] * values[1] +
values[2] * values[2],
).toDouble()
sumDelta += abs(magnitude - SensorManager.GRAVITY_EARTH.toDouble())
count += 1
if (count >= ACCELEROMETER_SAMPLE_TARGET) {
val result = AccelerometerSample(
samples = count,
averageDelta = sumDelta / count,
)
val token = cont.tryResume(result) ?: return
cont.completeResume(token)
sensorManager.unregisterListener(this)
}
}
override fun onAccuracyChanged(sensor: Sensor?, accuracy: Int) = Unit
}
val registered = sensorManager.registerListener(listener, sensor, SensorManager.SENSOR_DELAY_NORMAL)
if (!registered) {
cont.resume(null) { _, _, _ -> }
return@suspendCancellableCoroutine
}
cont.invokeOnCancellation { sensorManager.unregisterListener(listener) }
}
}
return sample
}
private fun classifyActivity(averageDelta: Double): String {
return when {
averageDelta <= 0.55 -> "stationary"
averageDelta <= 1.80 -> "walking"
else -> "running"
}
}
private fun classifyConfidence(samples: Int, averageDelta: Double): String {
if (samples < 6) return "low"
if (samples >= 14 && averageDelta > 0.4) return "high"
return "medium"
}
}
class MotionHandler private constructor(
private val appContext: Context,
private val dataSource: MotionDataSource,
) {
constructor(appContext: Context) : this(appContext = appContext, dataSource = SystemMotionDataSource)
suspend fun handleMotionActivity(paramsJson: String?): GatewaySession.InvokeResult {
if (!dataSource.hasPermission(appContext)) {
return GatewaySession.InvokeResult.error(
code = "MOTION_PERMISSION_REQUIRED",
message = "MOTION_PERMISSION_REQUIRED: grant Motion permission",
)
}
val request =
parseActivityRequest(paramsJson)
?: return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: expected JSON object",
)
return try {
val activity = dataSource.activity(appContext, request)
GatewaySession.InvokeResult.ok(
buildJsonObject {
put(
"activities",
buildJsonArray {
add(
buildJsonObject {
put("startISO", JsonPrimitive(activity.startISO))
put("endISO", JsonPrimitive(activity.endISO))
put("confidence", JsonPrimitive(activity.confidence))
put("isWalking", JsonPrimitive(activity.isWalking))
put("isRunning", JsonPrimitive(activity.isRunning))
put("isCycling", JsonPrimitive(activity.isCycling))
put("isAutomotive", JsonPrimitive(activity.isAutomotive))
put("isStationary", JsonPrimitive(activity.isStationary))
put("isUnknown", JsonPrimitive(activity.isUnknown))
},
)
},
)
}.toString(),
)
} catch (err: IllegalArgumentException) {
GatewaySession.InvokeResult.error(code = "MOTION_UNAVAILABLE", message = err.message ?: "MOTION_UNAVAILABLE")
} catch (err: Throwable) {
GatewaySession.InvokeResult.error(
code = "MOTION_UNAVAILABLE",
message = "MOTION_UNAVAILABLE: ${err.message ?: "motion activity failed"}",
)
}
}
suspend fun handleMotionPedometer(paramsJson: String?): GatewaySession.InvokeResult {
if (!dataSource.hasPermission(appContext)) {
return GatewaySession.InvokeResult.error(
code = "MOTION_PERMISSION_REQUIRED",
message = "MOTION_PERMISSION_REQUIRED: grant Motion permission",
)
}
val request =
parsePedometerRequest(paramsJson)
?: return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: expected JSON object",
)
return try {
val payload = dataSource.pedometer(appContext, request)
GatewaySession.InvokeResult.ok(
buildJsonObject {
put("startISO", JsonPrimitive(payload.startISO))
put("endISO", JsonPrimitive(payload.endISO))
payload.steps?.let { put("steps", JsonPrimitive(it)) }
payload.distanceMeters?.let { put("distanceMeters", JsonPrimitive(it)) }
payload.floorsAscended?.let { put("floorsAscended", JsonPrimitive(it)) }
payload.floorsDescended?.let { put("floorsDescended", JsonPrimitive(it)) }
}.toString(),
)
} catch (err: IllegalArgumentException) {
GatewaySession.InvokeResult.error(code = "MOTION_UNAVAILABLE", message = err.message ?: "MOTION_UNAVAILABLE")
} catch (err: Throwable) {
GatewaySession.InvokeResult.error(
code = "MOTION_UNAVAILABLE",
message = "MOTION_UNAVAILABLE: ${err.message ?: "pedometer query failed"}",
)
}
}
fun isAvailable(): Boolean = dataSource.isAvailable(appContext)
fun isActivityAvailable(): Boolean = dataSource.isActivityAvailable(appContext)
fun isPedometerAvailable(): Boolean = dataSource.isPedometerAvailable(appContext)
private fun parseActivityRequest(paramsJson: String?): MotionActivityRequest? {
if (paramsJson.isNullOrBlank()) {
return MotionActivityRequest(startISO = null, endISO = null, limit = 200)
}
val params =
try {
Json.parseToJsonElement(paramsJson).asObjectOrNull()
} catch (_: Throwable) {
null
} ?: return null
val limit = ((params["limit"] as? JsonPrimitive)?.content?.toIntOrNull() ?: 200).coerceIn(1, 1000)
return MotionActivityRequest(
startISO = (params["startISO"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
endISO = (params["endISO"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
limit = limit,
)
}
private fun parsePedometerRequest(paramsJson: String?): MotionPedometerRequest? {
if (paramsJson.isNullOrBlank()) {
return MotionPedometerRequest(startISO = null, endISO = null)
}
val params =
try {
Json.parseToJsonElement(paramsJson).asObjectOrNull()
} catch (_: Throwable) {
null
} ?: return null
return MotionPedometerRequest(
startISO = (params["startISO"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
endISO = (params["endISO"] as? JsonPrimitive)?.content?.trim()?.ifEmpty { null },
)
}
companion object {
fun isMotionCapabilityAvailable(context: Context): Boolean = SystemMotionDataSource.isAvailable(context)
internal fun forTesting(
appContext: Context,
dataSource: MotionDataSource,
): MotionHandler = MotionHandler(appContext = appContext, dataSource = dataSource)
}
}

View file

@ -0,0 +1,84 @@
package ai.openclaw.app.node
import ai.openclaw.app.gateway.parseInvokeErrorFromThrowable
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonElement
import kotlinx.serialization.json.JsonNull
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.contentOrNull
const val DEFAULT_SEAM_COLOR_ARGB: Long = 0xFF4F7A9A
data class Quad<A, B, C, D>(val first: A, val second: B, val third: C, val fourth: D)
fun String.toJsonString(): String {
val escaped =
this.replace("\\", "\\\\")
.replace("\"", "\\\"")
.replace("\n", "\\n")
.replace("\r", "\\r")
return "\"$escaped\""
}
fun JsonElement?.asObjectOrNull(): JsonObject? = this as? JsonObject
fun parseJsonParamsObject(paramsJson: String?): JsonObject? {
if (paramsJson.isNullOrBlank()) return null
return try {
Json.parseToJsonElement(paramsJson).asObjectOrNull()
} catch (_: Throwable) {
null
}
}
fun readJsonPrimitive(params: JsonObject?, key: String): JsonPrimitive? = params?.get(key) as? JsonPrimitive
fun parseJsonInt(params: JsonObject?, key: String): Int? =
readJsonPrimitive(params, key)?.contentOrNull?.toIntOrNull()
fun parseJsonDouble(params: JsonObject?, key: String): Double? =
readJsonPrimitive(params, key)?.contentOrNull?.toDoubleOrNull()
fun parseJsonString(params: JsonObject?, key: String): String? =
readJsonPrimitive(params, key)?.contentOrNull
fun parseJsonBooleanFlag(params: JsonObject?, key: String): Boolean? {
val value = readJsonPrimitive(params, key)?.contentOrNull?.trim()?.lowercase() ?: return null
return when (value) {
"true" -> true
"false" -> false
else -> null
}
}
fun JsonElement?.asStringOrNull(): String? =
when (this) {
is JsonNull -> null
is JsonPrimitive -> content
else -> null
}
fun parseHexColorArgb(raw: String?): Long? {
val trimmed = raw?.trim().orEmpty()
if (trimmed.isEmpty()) return null
val hex = if (trimmed.startsWith("#")) trimmed.drop(1) else trimmed
if (hex.length != 6) return null
val rgb = hex.toLongOrNull(16) ?: return null
return 0xFF000000L or rgb
}
fun invokeErrorFromThrowable(err: Throwable): Pair<String, String> {
val parsed = parseInvokeErrorFromThrowable(err, fallbackMessage = "UNAVAILABLE: error")
val message = if (parsed.hadExplicitCode) parsed.prefixedMessage else parsed.message
return parsed.code to message
}
fun normalizeMainKey(raw: String?): String? {
val trimmed = raw?.trim().orEmpty()
return if (trimmed.isEmpty()) null else trimmed
}
fun isCanonicalMainSessionKey(key: String): Boolean {
return key == "main"
}

View file

@ -0,0 +1,161 @@
package ai.openclaw.app.node
import android.content.Context
import ai.openclaw.app.gateway.GatewaySession
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonArray
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.buildJsonObject
import kotlinx.serialization.json.contentOrNull
import kotlinx.serialization.json.put
internal interface NotificationsStateProvider {
fun readSnapshot(context: Context): DeviceNotificationSnapshot
fun requestServiceRebind(context: Context)
fun executeAction(context: Context, request: NotificationActionRequest): NotificationActionResult
}
private object SystemNotificationsStateProvider : NotificationsStateProvider {
override fun readSnapshot(context: Context): DeviceNotificationSnapshot {
val enabled = DeviceNotificationListenerService.isAccessEnabled(context)
if (!enabled) {
return DeviceNotificationSnapshot(
enabled = false,
connected = false,
notifications = emptyList(),
)
}
return DeviceNotificationListenerService.snapshot(context, enabled = true)
}
override fun requestServiceRebind(context: Context) {
DeviceNotificationListenerService.requestServiceRebind(context)
}
override fun executeAction(context: Context, request: NotificationActionRequest): NotificationActionResult {
return DeviceNotificationListenerService.executeAction(context, request)
}
}
class NotificationsHandler private constructor(
private val appContext: Context,
private val stateProvider: NotificationsStateProvider,
) {
constructor(appContext: Context) : this(appContext = appContext, stateProvider = SystemNotificationsStateProvider)
suspend fun handleNotificationsList(_paramsJson: String?): GatewaySession.InvokeResult {
val snapshot = readSnapshotWithRebind()
return GatewaySession.InvokeResult.ok(snapshotPayloadJson(snapshot))
}
suspend fun handleNotificationsActions(paramsJson: String?): GatewaySession.InvokeResult {
readSnapshotWithRebind()
val params = parseParamsObject(paramsJson)
?: return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: expected JSON object",
)
val key =
readString(params, "key")
?: return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: key required",
)
val actionRaw =
readString(params, "action")?.lowercase()
?: return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: action required (open|dismiss|reply)",
)
val action =
when (actionRaw) {
"open" -> NotificationActionKind.Open
"dismiss" -> NotificationActionKind.Dismiss
"reply" -> NotificationActionKind.Reply
else ->
return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: action must be open|dismiss|reply",
)
}
val replyText = readString(params, "replyText")
if (action == NotificationActionKind.Reply && replyText.isNullOrBlank()) {
return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: replyText required for reply action",
)
}
val result =
stateProvider.executeAction(
appContext,
NotificationActionRequest(
key = key,
kind = action,
replyText = replyText,
),
)
if (!result.ok) {
return GatewaySession.InvokeResult.error(
code = result.code ?: "UNAVAILABLE",
message = result.message ?: "notification action failed",
)
}
val payload =
buildJsonObject {
put("ok", JsonPrimitive(true))
put("key", JsonPrimitive(key))
put("action", JsonPrimitive(actionRaw))
}.toString()
return GatewaySession.InvokeResult.ok(payload)
}
private fun readSnapshotWithRebind(): DeviceNotificationSnapshot {
val snapshot = stateProvider.readSnapshot(appContext)
if (snapshot.enabled && !snapshot.connected) {
stateProvider.requestServiceRebind(appContext)
}
return snapshot
}
private fun snapshotPayloadJson(snapshot: DeviceNotificationSnapshot): String {
return buildJsonObject {
put("enabled", JsonPrimitive(snapshot.enabled))
put("connected", JsonPrimitive(snapshot.connected))
put("count", JsonPrimitive(snapshot.notifications.size))
put(
"notifications",
JsonArray(
snapshot.notifications.map { entry -> entry.toJsonObject() },
),
)
}.toString()
}
private fun parseParamsObject(paramsJson: String?): JsonObject? {
if (paramsJson.isNullOrBlank()) return null
return try {
Json.parseToJsonElement(paramsJson).asObjectOrNull()
} catch (_: Throwable) {
null
}
}
private fun readString(params: JsonObject, key: String): String? =
(params[key] as? JsonPrimitive)
?.contentOrNull
?.trim()
?.takeIf { it.isNotEmpty() }
companion object {
internal fun forTesting(
appContext: Context,
stateProvider: NotificationsStateProvider,
): NotificationsHandler = NotificationsHandler(appContext = appContext, stateProvider = stateProvider)
}
}

View file

@ -0,0 +1,303 @@
package ai.openclaw.app.node
import android.Manifest
import android.content.ContentResolver
import android.content.ContentUris
import android.content.Context
import android.graphics.Bitmap
import android.graphics.BitmapFactory
import android.net.Uri
import android.os.Build
import android.os.Bundle
import android.provider.MediaStore
import androidx.core.content.ContextCompat
import androidx.core.graphics.scale
import ai.openclaw.app.gateway.GatewaySession
import java.io.ByteArrayOutputStream
import java.time.Instant
import kotlin.math.max
import kotlin.math.roundToInt
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.buildJsonArray
import kotlinx.serialization.json.buildJsonObject
import kotlinx.serialization.json.put
private const val DEFAULT_PHOTOS_LIMIT = 1
private const val DEFAULT_PHOTOS_MAX_WIDTH = 1600
private const val DEFAULT_PHOTOS_QUALITY = 0.85
private const val MAX_TOTAL_BASE64_CHARS = 340 * 1024
private const val MAX_PER_PHOTO_BASE64_CHARS = 300 * 1024
internal data class PhotosLatestRequest(
val limit: Int,
val maxWidth: Int,
val quality: Double,
)
internal data class EncodedPhotoPayload(
val format: String,
val base64: String,
val width: Int,
val height: Int,
val createdAt: String?,
)
internal interface PhotosDataSource {
fun hasPermission(context: Context): Boolean
fun latest(context: Context, request: PhotosLatestRequest): List<EncodedPhotoPayload>
}
private object SystemPhotosDataSource : PhotosDataSource {
override fun hasPermission(context: Context): Boolean {
val permission =
if (Build.VERSION.SDK_INT >= 33) {
Manifest.permission.READ_MEDIA_IMAGES
} else {
Manifest.permission.READ_EXTERNAL_STORAGE
}
return ContextCompat.checkSelfPermission(context, permission) == android.content.pm.PackageManager.PERMISSION_GRANTED
}
override fun latest(context: Context, request: PhotosLatestRequest): List<EncodedPhotoPayload> {
val resolver = context.contentResolver
val rows = queryLatestRows(resolver, request.limit)
if (rows.isEmpty()) return emptyList()
var remainingBudget = MAX_TOTAL_BASE64_CHARS
val out = mutableListOf<EncodedPhotoPayload>()
for (row in rows) {
if (remainingBudget <= 0) break
val bitmap = decodeScaledBitmap(resolver, row.uri, request.maxWidth) ?: continue
try {
val encoded = encodeJpegUnderBudget(bitmap, request.quality, MAX_PER_PHOTO_BASE64_CHARS)
if (encoded == null) continue
if (encoded.base64.length > remainingBudget) break
remainingBudget -= encoded.base64.length
out +=
EncodedPhotoPayload(
format = "jpeg",
base64 = encoded.base64,
width = encoded.width,
height = encoded.height,
createdAt = row.createdAtMs?.let { Instant.ofEpochMilli(it).toString() },
)
} finally {
bitmap.recycle()
}
}
return out
}
private data class PhotoRow(
val uri: Uri,
val createdAtMs: Long?,
)
private data class EncodedJpeg(
val base64: String,
val width: Int,
val height: Int,
)
private fun queryLatestRows(resolver: ContentResolver, limit: Int): List<PhotoRow> {
val projection =
arrayOf(
MediaStore.Images.Media._ID,
MediaStore.Images.Media.DATE_TAKEN,
MediaStore.Images.Media.DATE_ADDED,
)
val sortOrder =
"${MediaStore.Images.Media.DATE_TAKEN} DESC, ${MediaStore.Images.Media.DATE_ADDED} DESC"
val args =
Bundle().apply {
putString(ContentResolver.QUERY_ARG_SQL_SORT_ORDER, sortOrder)
putInt(ContentResolver.QUERY_ARG_LIMIT, limit)
}
resolver.query(
MediaStore.Images.Media.EXTERNAL_CONTENT_URI,
projection,
args,
null,
).use { cursor ->
if (cursor == null) return emptyList()
val idIndex = cursor.getColumnIndexOrThrow(MediaStore.Images.Media._ID)
val takenIndex = cursor.getColumnIndexOrThrow(MediaStore.Images.Media.DATE_TAKEN)
val addedIndex = cursor.getColumnIndexOrThrow(MediaStore.Images.Media.DATE_ADDED)
val rows = mutableListOf<PhotoRow>()
while (cursor.moveToNext()) {
val id = cursor.getLong(idIndex)
val takenMs = cursor.getLong(takenIndex).takeIf { it > 0L }
val addedMs = cursor.getLong(addedIndex).takeIf { it > 0L }?.times(1000L)
rows +=
PhotoRow(
uri = ContentUris.withAppendedId(MediaStore.Images.Media.EXTERNAL_CONTENT_URI, id),
createdAtMs = takenMs ?: addedMs,
)
}
return rows
}
}
private fun decodeScaledBitmap(
resolver: ContentResolver,
uri: Uri,
maxWidth: Int,
): Bitmap? {
val bounds = BitmapFactory.Options().apply { inJustDecodeBounds = true }
resolver.openInputStream(uri).use { input ->
if (input == null) return null
BitmapFactory.decodeStream(input, null, bounds)
}
if (bounds.outWidth <= 0 || bounds.outHeight <= 0) return null
val inSampleSize = computeInSampleSize(bounds.outWidth, maxWidth)
val decodeOptions = BitmapFactory.Options().apply { this.inSampleSize = inSampleSize }
val decoded =
resolver.openInputStream(uri).use { input ->
if (input == null) return null
BitmapFactory.decodeStream(input, null, decodeOptions)
} ?: return null
if (decoded.width <= maxWidth) return decoded
val targetHeight = max(1, ((decoded.height.toDouble() * maxWidth) / decoded.width).roundToInt())
return try {
decoded.scale(maxWidth, targetHeight, true)
} finally {
decoded.recycle()
}
}
private fun computeInSampleSize(width: Int, maxWidth: Int): Int {
var sample = 1
var candidate = width
while (candidate > maxWidth && sample < 64) {
sample *= 2
candidate = width / sample
}
return sample
}
private fun encodeJpegUnderBudget(
bitmap: Bitmap,
quality: Double,
maxBase64Chars: Int,
): EncodedJpeg? {
var working = bitmap
try {
var jpegQuality = (quality.coerceIn(0.1, 1.0) * 100.0).roundToInt().coerceIn(10, 100)
repeat(10) {
val out = ByteArrayOutputStream()
val ok = working.compress(Bitmap.CompressFormat.JPEG, jpegQuality, out)
if (!ok) return null
val bytes = out.toByteArray()
val base64 = android.util.Base64.encodeToString(bytes, android.util.Base64.NO_WRAP)
if (base64.length <= maxBase64Chars) {
return EncodedJpeg(
base64 = base64,
width = working.width,
height = working.height,
)
}
if (jpegQuality > 35) {
jpegQuality = max(25, jpegQuality - 15)
return@repeat
}
val nextWidth = max(240, (working.width * 0.75f).roundToInt())
if (nextWidth >= working.width) return null
val nextHeight = max(1, ((working.height.toDouble() * nextWidth) / working.width).roundToInt())
val previous = working
working = working.scale(nextWidth, nextHeight, true)
if (previous !== bitmap) previous.recycle()
}
return null
} finally {
if (working !== bitmap) working.recycle()
}
}
}
class PhotosHandler private constructor(
private val appContext: Context,
private val dataSource: PhotosDataSource,
) {
constructor(appContext: Context) : this(appContext = appContext, dataSource = SystemPhotosDataSource)
fun handlePhotosLatest(paramsJson: String?): GatewaySession.InvokeResult {
if (!dataSource.hasPermission(appContext)) {
return GatewaySession.InvokeResult.error(
code = "PHOTOS_PERMISSION_REQUIRED",
message = "PHOTOS_PERMISSION_REQUIRED: grant Photos permission",
)
}
val request =
parseRequest(paramsJson)
?: return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: expected JSON object",
)
return try {
val photos = dataSource.latest(appContext, request)
val payload =
buildJsonObject {
put(
"photos",
buildJsonArray {
photos.forEach { photo ->
add(
buildJsonObject {
put("format", JsonPrimitive(photo.format))
put("base64", JsonPrimitive(photo.base64))
put("width", JsonPrimitive(photo.width))
put("height", JsonPrimitive(photo.height))
photo.createdAt?.let { put("createdAt", JsonPrimitive(it)) }
},
)
}
},
)
}.toString()
GatewaySession.InvokeResult.ok(payload)
} catch (err: Throwable) {
GatewaySession.InvokeResult.error(
code = "PHOTOS_UNAVAILABLE",
message = "PHOTOS_UNAVAILABLE: ${err.message ?: "photo fetch failed"}",
)
}
}
private fun parseRequest(paramsJson: String?): PhotosLatestRequest? {
if (paramsJson.isNullOrBlank()) {
return PhotosLatestRequest(
limit = DEFAULT_PHOTOS_LIMIT,
maxWidth = DEFAULT_PHOTOS_MAX_WIDTH,
quality = DEFAULT_PHOTOS_QUALITY,
)
}
val params =
try {
Json.parseToJsonElement(paramsJson).asObjectOrNull()
} catch (_: Throwable) {
null
} ?: return null
val limitRaw = (params["limit"] as? JsonPrimitive)?.content?.toIntOrNull()
val maxWidthRaw = (params["maxWidth"] as? JsonPrimitive)?.content?.toIntOrNull()
val qualityRaw = (params["quality"] as? JsonPrimitive)?.content?.toDoubleOrNull()
val limit = (limitRaw ?: DEFAULT_PHOTOS_LIMIT).coerceIn(1, 20)
val maxWidth = (maxWidthRaw ?: DEFAULT_PHOTOS_MAX_WIDTH).coerceIn(240, 4096)
val quality = (qualityRaw ?: DEFAULT_PHOTOS_QUALITY).coerceIn(0.1, 1.0)
return PhotosLatestRequest(limit = limit, maxWidth = maxWidth, quality = quality)
}
companion object {
internal fun forTesting(
appContext: Context,
dataSource: PhotosDataSource,
): PhotosHandler = PhotosHandler(appContext = appContext, dataSource = dataSource)
}
}

View file

@ -0,0 +1,36 @@
package ai.openclaw.app.node
import ai.openclaw.app.gateway.GatewaySession
class SmsHandler(
private val sms: SmsManager,
) {
suspend fun handleSmsSend(paramsJson: String?): GatewaySession.InvokeResult {
val res = sms.send(paramsJson)
if (res.ok) {
return GatewaySession.InvokeResult.ok(res.payloadJson)
}
return errorResult(res.error, defaultCode = "SMS_SEND_FAILED")
}
suspend fun handleSmsSearch(paramsJson: String?): GatewaySession.InvokeResult {
val res = sms.search(paramsJson)
if (res.ok) {
return GatewaySession.InvokeResult.ok(res.payloadJson)
}
return errorResult(res.error, defaultCode = "SMS_SEARCH_FAILED")
}
private fun errorResult(error: String?, defaultCode: String): GatewaySession.InvokeResult {
val rawMessage = error ?: defaultCode
val idx = rawMessage.indexOf(':')
val code = if (idx > 0) rawMessage.substring(0, idx).trim() else defaultCode
val message =
if (idx > 0 && code == rawMessage.substring(0, idx).trim()) {
rawMessage.substring(idx + 1).trim().ifEmpty { rawMessage }
} else {
rawMessage
}
return GatewaySession.InvokeResult.error(code = code, message = message)
}
}

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,172 @@
package ai.openclaw.app.node
import android.Manifest
import android.app.NotificationChannel
import android.app.NotificationManager
import android.content.Context
import android.content.pm.PackageManager
import android.os.Build
import androidx.core.app.NotificationCompat
import androidx.core.app.NotificationManagerCompat
import androidx.core.content.ContextCompat
import ai.openclaw.app.gateway.GatewaySession
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.contentOrNull
private const val NOTIFICATION_CHANNEL_BASE_ID = "openclaw.system.notify"
internal data class SystemNotifyRequest(
val title: String,
val body: String,
val sound: String?,
val priority: String?,
)
internal interface SystemNotificationPoster {
fun isAuthorized(): Boolean
fun post(request: SystemNotifyRequest)
}
private class AndroidSystemNotificationPoster(
private val appContext: Context,
) : SystemNotificationPoster {
override fun isAuthorized(): Boolean {
if (Build.VERSION.SDK_INT >= 33) {
val granted =
ContextCompat.checkSelfPermission(appContext, Manifest.permission.POST_NOTIFICATIONS) ==
PackageManager.PERMISSION_GRANTED
if (!granted) return false
}
return NotificationManagerCompat.from(appContext).areNotificationsEnabled()
}
override fun post(request: SystemNotifyRequest) {
val channelId = ensureChannel(request.priority)
val silent = isSilentSound(request.sound)
val notification =
NotificationCompat.Builder(appContext, channelId)
.setSmallIcon(android.R.drawable.ic_dialog_info)
.setContentTitle(request.title)
.setContentText(request.body)
.setPriority(compatPriority(request.priority))
.setAutoCancel(true)
.setOnlyAlertOnce(true)
.setSilent(silent)
.build()
if (
Build.VERSION.SDK_INT >= 33 &&
ContextCompat.checkSelfPermission(appContext, Manifest.permission.POST_NOTIFICATIONS) !=
PackageManager.PERMISSION_GRANTED
) {
throw SecurityException("notifications permission missing")
}
NotificationManagerCompat.from(appContext).notify((System.currentTimeMillis() and 0x7FFFFFFF).toInt(), notification)
}
private fun ensureChannel(priority: String?): String {
val normalizedPriority = priority.orEmpty().trim().lowercase()
val (suffix, importance, name) =
when (normalizedPriority) {
"passive" -> Triple("passive", NotificationManager.IMPORTANCE_LOW, "OpenClaw Passive")
"timesensitive" -> Triple("timesensitive", NotificationManager.IMPORTANCE_HIGH, "OpenClaw Time Sensitive")
else -> Triple("active", NotificationManager.IMPORTANCE_DEFAULT, "OpenClaw Active")
}
val channelId = "$NOTIFICATION_CHANNEL_BASE_ID.$suffix"
val manager = appContext.getSystemService(NotificationManager::class.java)
val existing = manager.getNotificationChannel(channelId)
if (existing == null) {
manager.createNotificationChannel(NotificationChannel(channelId, name, importance))
}
return channelId
}
private fun compatPriority(priority: String?): Int {
return when (priority.orEmpty().trim().lowercase()) {
"passive" -> NotificationCompat.PRIORITY_LOW
"timesensitive" -> NotificationCompat.PRIORITY_HIGH
else -> NotificationCompat.PRIORITY_DEFAULT
}
}
private fun isSilentSound(sound: String?): Boolean {
val normalized = sound?.trim()?.lowercase() ?: return false
return normalized in setOf("none", "silent", "off", "false", "0")
}
}
class SystemHandler private constructor(
private val poster: SystemNotificationPoster,
) {
constructor(appContext: Context) : this(poster = AndroidSystemNotificationPoster(appContext))
fun handleSystemNotify(paramsJson: String?): GatewaySession.InvokeResult {
val params =
parseNotifyRequest(paramsJson)
?: return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: expected JSON object with title/body",
)
if (params.title.isEmpty() && params.body.isEmpty()) {
return GatewaySession.InvokeResult.error(
code = "INVALID_REQUEST",
message = "INVALID_REQUEST: empty notification",
)
}
if (!poster.isAuthorized()) {
return GatewaySession.InvokeResult.error(
code = "NOT_AUTHORIZED",
message = "NOT_AUTHORIZED: notifications",
)
}
return try {
poster.post(params)
GatewaySession.InvokeResult.ok(null)
} catch (_: SecurityException) {
GatewaySession.InvokeResult.error(
code = "NOT_AUTHORIZED",
message = "NOT_AUTHORIZED: notifications",
)
} catch (err: Throwable) {
GatewaySession.InvokeResult.error(
code = "UNAVAILABLE",
message = "NOTIFICATION_FAILED: ${err.message ?: "notification post failed"}",
)
}
}
private fun parseNotifyRequest(paramsJson: String?): SystemNotifyRequest? {
val params = parseParamsObject(paramsJson) ?: return null
val rawTitle =
(params["title"] as? JsonPrimitive)
?.contentOrNull
?: return null
val rawBody =
(params["body"] as? JsonPrimitive)
?.contentOrNull
?: return null
val sound = (params["sound"] as? JsonPrimitive)?.contentOrNull
val priority = (params["priority"] as? JsonPrimitive)?.contentOrNull
return SystemNotifyRequest(
title = rawTitle.trim(),
body = rawBody.trim(),
sound = sound?.trim()?.ifEmpty { null },
priority = priority?.trim()?.ifEmpty { null },
)
}
private fun parseParamsObject(paramsJson: String?): JsonObject? {
if (paramsJson.isNullOrBlank()) return null
return try {
Json.parseToJsonElement(paramsJson).asObjectOrNull()
} catch (_: Throwable) {
null
}
}
companion object {
internal fun forTesting(poster: SystemNotificationPoster): SystemHandler = SystemHandler(poster)
}
}

View file

@ -0,0 +1,69 @@
package ai.openclaw.app.protocol
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
object OpenClawCanvasA2UIAction {
fun extractActionName(userAction: JsonObject): String? {
val name =
(userAction["name"] as? JsonPrimitive)
?.content
?.trim()
.orEmpty()
if (name.isNotEmpty()) return name
val action =
(userAction["action"] as? JsonPrimitive)
?.content
?.trim()
.orEmpty()
return action.ifEmpty { null }
}
fun sanitizeTagValue(value: String): String {
val trimmed = value.trim().ifEmpty { "-" }
val normalized = trimmed.replace(" ", "_")
val out = StringBuilder(normalized.length)
for (c in normalized) {
val ok =
c.isLetterOrDigit() ||
c == '_' ||
c == '-' ||
c == '.' ||
c == ':'
out.append(if (ok) c else '_')
}
return out.toString()
}
fun formatAgentMessage(
actionName: String,
sessionKey: String,
surfaceId: String,
sourceComponentId: String,
host: String,
instanceId: String,
contextJson: String?,
): String {
val ctxSuffix = contextJson?.takeIf { it.isNotBlank() }?.let { " ctx=$it" }.orEmpty()
return listOf(
"CANVAS_A2UI",
"action=${sanitizeTagValue(actionName)}",
"session=${sanitizeTagValue(sessionKey)}",
"surface=${sanitizeTagValue(surfaceId)}",
"component=${sanitizeTagValue(sourceComponentId)}",
"host=${sanitizeTagValue(host)}",
"instance=${sanitizeTagValue(instanceId)}$ctxSuffix",
"default=update_canvas",
).joinToString(separator = " ")
}
fun jsDispatchA2UIActionStatus(actionId: String, ok: Boolean, error: String?): String {
val err = jsonStringLiteral(error ?: "")
val okLiteral = if (ok) "true" else "false"
val idLiteral = jsonStringLiteral(actionId)
return "window.dispatchEvent(new CustomEvent('openclaw:a2ui-action-status', { detail: { id: ${idLiteral}, ok: ${okLiteral}, error: ${err} } }));"
}
private fun jsonStringLiteral(raw: String): String =
JsonPrimitive(raw).toString().replace("\u2028", "\\u2028").replace("\u2029", "\\u2029")
}

View file

@ -0,0 +1,150 @@
package ai.openclaw.app.protocol
enum class OpenClawCapability(val rawValue: String) {
Canvas("canvas"),
Camera("camera"),
Sms("sms"),
VoiceWake("voiceWake"),
Location("location"),
Device("device"),
Notifications("notifications"),
System("system"),
Photos("photos"),
Contacts("contacts"),
Calendar("calendar"),
Motion("motion"),
CallLog("callLog"),
}
enum class OpenClawCanvasCommand(val rawValue: String) {
Present("canvas.present"),
Hide("canvas.hide"),
Navigate("canvas.navigate"),
Eval("canvas.eval"),
Snapshot("canvas.snapshot"),
;
companion object {
const val NamespacePrefix: String = "canvas."
}
}
enum class OpenClawCanvasA2UICommand(val rawValue: String) {
Push("canvas.a2ui.push"),
PushJSONL("canvas.a2ui.pushJSONL"),
Reset("canvas.a2ui.reset"),
;
companion object {
const val NamespacePrefix: String = "canvas.a2ui."
}
}
enum class OpenClawCameraCommand(val rawValue: String) {
List("camera.list"),
Snap("camera.snap"),
Clip("camera.clip"),
;
companion object {
const val NamespacePrefix: String = "camera."
}
}
enum class OpenClawSmsCommand(val rawValue: String) {
Send("sms.send"),
Search("sms.search"),
;
companion object {
const val NamespacePrefix: String = "sms."
}
}
enum class OpenClawLocationCommand(val rawValue: String) {
Get("location.get"),
;
companion object {
const val NamespacePrefix: String = "location."
}
}
enum class OpenClawDeviceCommand(val rawValue: String) {
Status("device.status"),
Info("device.info"),
Permissions("device.permissions"),
Health("device.health"),
;
companion object {
const val NamespacePrefix: String = "device."
}
}
enum class OpenClawNotificationsCommand(val rawValue: String) {
List("notifications.list"),
Actions("notifications.actions"),
;
companion object {
const val NamespacePrefix: String = "notifications."
}
}
enum class OpenClawSystemCommand(val rawValue: String) {
Notify("system.notify"),
;
companion object {
const val NamespacePrefix: String = "system."
}
}
enum class OpenClawPhotosCommand(val rawValue: String) {
Latest("photos.latest"),
;
companion object {
const val NamespacePrefix: String = "photos."
}
}
enum class OpenClawContactsCommand(val rawValue: String) {
Search("contacts.search"),
Add("contacts.add"),
;
companion object {
const val NamespacePrefix: String = "contacts."
}
}
enum class OpenClawCalendarCommand(val rawValue: String) {
Events("calendar.events"),
Add("calendar.add"),
;
companion object {
const val NamespacePrefix: String = "calendar."
}
}
enum class OpenClawMotionCommand(val rawValue: String) {
Activity("motion.activity"),
Pedometer("motion.pedometer"),
;
companion object {
const val NamespacePrefix: String = "motion."
}
}
enum class OpenClawCallLogCommand(val rawValue: String) {
Search("callLog.search"),
;
companion object {
const val NamespacePrefix: String = "callLog."
}
}

View file

@ -0,0 +1,222 @@
package ai.openclaw.app.tools
import android.content.Context
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonArray
import kotlinx.serialization.json.JsonElement
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.contentOrNull
@Serializable
private data class ToolDisplayActionSpec(
val label: String? = null,
val detailKeys: List<String>? = null,
)
@Serializable
private data class ToolDisplaySpec(
val emoji: String? = null,
val title: String? = null,
val label: String? = null,
val detailKeys: List<String>? = null,
val actions: Map<String, ToolDisplayActionSpec>? = null,
)
@Serializable
private data class ToolDisplayConfig(
val version: Int? = null,
val fallback: ToolDisplaySpec? = null,
val tools: Map<String, ToolDisplaySpec>? = null,
)
data class ToolDisplaySummary(
val name: String,
val emoji: String,
val title: String,
val label: String,
val verb: String?,
val detail: String?,
) {
val detailLine: String?
get() {
val parts = mutableListOf<String>()
if (!verb.isNullOrBlank()) parts.add(verb)
if (!detail.isNullOrBlank()) parts.add(detail)
return if (parts.isEmpty()) null else parts.joinToString(" · ")
}
val summaryLine: String
get() = if (detailLine != null) "${emoji} ${label}: ${detailLine}" else "${emoji} ${label}"
}
object ToolDisplayRegistry {
private const val CONFIG_ASSET = "tool-display.json"
private val json = Json { ignoreUnknownKeys = true }
@Volatile private var cachedConfig: ToolDisplayConfig? = null
fun resolve(
context: Context,
name: String?,
args: JsonObject?,
meta: String? = null,
): ToolDisplaySummary {
val trimmedName = name?.trim().orEmpty().ifEmpty { "tool" }
val key = trimmedName.lowercase()
val config = loadConfig(context)
val spec = config.tools?.get(key)
val fallback = config.fallback
val emoji = spec?.emoji ?: fallback?.emoji ?: "🧩"
val title = spec?.title ?: titleFromName(trimmedName)
val label = spec?.label ?: trimmedName
val actionRaw = args?.get("action")?.asStringOrNull()?.trim()
val action = actionRaw?.takeIf { it.isNotEmpty() }
val actionSpec = action?.let { spec?.actions?.get(it) }
val verb = normalizeVerb(actionSpec?.label ?: action)
var detail: String? = null
if (key == "read") {
detail = readDetail(args)
} else if (key == "write" || key == "edit" || key == "attach") {
detail = pathDetail(args)
}
val detailKeys = actionSpec?.detailKeys ?: spec?.detailKeys ?: fallback?.detailKeys ?: emptyList()
if (detail == null) {
detail = firstValue(args, detailKeys)
}
if (detail == null) {
detail = meta
}
if (detail != null) {
detail = shortenHomeInString(detail)
}
return ToolDisplaySummary(
name = trimmedName,
emoji = emoji,
title = title,
label = label,
verb = verb,
detail = detail,
)
}
private fun loadConfig(context: Context): ToolDisplayConfig {
val existing = cachedConfig
if (existing != null) return existing
return try {
val jsonString = context.assets.open(CONFIG_ASSET).bufferedReader().use { it.readText() }
val decoded = json.decodeFromString(ToolDisplayConfig.serializer(), jsonString)
cachedConfig = decoded
decoded
} catch (_: Throwable) {
val fallback = ToolDisplayConfig()
cachedConfig = fallback
fallback
}
}
private fun titleFromName(name: String): String {
val cleaned = name.replace("_", " ").trim()
if (cleaned.isEmpty()) return "Tool"
return cleaned
.split(Regex("\\s+"))
.joinToString(" ") { part ->
val upper = part.uppercase()
if (part.length <= 2 && part == upper) part
else upper.firstOrNull()?.toString().orEmpty() + part.lowercase().drop(1)
}
}
private fun normalizeVerb(value: String?): String? {
val trimmed = value?.trim().orEmpty()
if (trimmed.isEmpty()) return null
return trimmed.replace("_", " ")
}
private fun readDetail(args: JsonObject?): String? {
val path = args?.get("path")?.asStringOrNull() ?: return null
val offset = args["offset"].asNumberOrNull()
val limit = args["limit"].asNumberOrNull()
return if (offset != null && limit != null) {
val end = offset + limit
"${path}:${offset.toInt()}-${end.toInt()}"
} else {
path
}
}
private fun pathDetail(args: JsonObject?): String? {
return args?.get("path")?.asStringOrNull()
}
private fun firstValue(args: JsonObject?, keys: List<String>): String? {
for (key in keys) {
val value = valueForPath(args, key)
val rendered = renderValue(value)
if (!rendered.isNullOrBlank()) return rendered
}
return null
}
private fun valueForPath(args: JsonObject?, path: String): JsonElement? {
var current: JsonElement? = args
for (segment in path.split(".")) {
if (segment.isBlank()) return null
val obj = current as? JsonObject ?: return null
current = obj[segment]
}
return current
}
private fun renderValue(value: JsonElement?): String? {
if (value == null) return null
if (value is JsonPrimitive) {
if (value.isString) {
val trimmed = value.contentOrNull?.trim().orEmpty()
if (trimmed.isEmpty()) return null
val firstLine = trimmed.lineSequence().firstOrNull()?.trim().orEmpty()
if (firstLine.isEmpty()) return null
return if (firstLine.length > 160) "${firstLine.take(157)}…" else firstLine
}
val raw = value.contentOrNull?.trim().orEmpty()
raw.toBooleanStrictOrNull()?.let { return it.toString() }
raw.toLongOrNull()?.let { return it.toString() }
raw.toDoubleOrNull()?.let { return it.toString() }
}
if (value is JsonArray) {
val items = value.mapNotNull { renderValue(it) }
if (items.isEmpty()) return null
val preview = items.take(3).joinToString(", ")
return if (items.size > 3) "${preview}…" else preview
}
return null
}
private fun shortenHomeInString(value: String): String {
val home = System.getProperty("user.home")?.takeIf { it.isNotBlank() }
?: System.getenv("HOME")?.takeIf { it.isNotBlank() }
if (home.isNullOrEmpty()) return value
return value.replace(home, "~")
.replace(Regex("/Users/[^/]+"), "~")
.replace(Regex("/home/[^/]+"), "~")
}
private fun JsonElement?.asStringOrNull(): String? {
val primitive = this as? JsonPrimitive ?: return null
return if (primitive.isString) primitive.contentOrNull else primitive.toString()
}
private fun JsonElement?.asNumberOrNull(): Double? {
val primitive = this as? JsonPrimitive ?: return null
val raw = primitive.contentOrNull ?: return null
return raw.toDoubleOrNull()
}
}

View file

@ -0,0 +1,44 @@
package ai.openclaw.app.ui
import androidx.compose.foundation.background
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.runtime.Composable
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableFloatStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.setValue
import androidx.compose.ui.Modifier
import androidx.compose.ui.draw.alpha
import androidx.compose.ui.graphics.Color
import kotlinx.coroutines.delay
@Composable
fun CameraFlashOverlay(
token: Long,
modifier: Modifier = Modifier,
) {
Box(modifier = modifier.fillMaxSize()) {
CameraFlash(token = token)
}
}
@Composable
private fun CameraFlash(token: Long) {
var alpha by remember { mutableFloatStateOf(0f) }
LaunchedEffect(token) {
if (token == 0L) return@LaunchedEffect
alpha = 0.85f
delay(110)
alpha = 0f
}
Box(
modifier =
Modifier
.fillMaxSize()
.alpha(alpha)
.background(Color.White),
)
}

View file

@ -0,0 +1,175 @@
package ai.openclaw.app.ui
import android.annotation.SuppressLint
import android.util.Log
import android.view.View
import android.webkit.ConsoleMessage
import android.webkit.JavascriptInterface
import android.webkit.WebChromeClient
import android.webkit.WebResourceError
import android.webkit.WebResourceRequest
import android.webkit.WebResourceResponse
import android.webkit.WebSettings
import android.webkit.WebView
import android.webkit.WebViewClient
import androidx.compose.runtime.Composable
import androidx.compose.runtime.DisposableEffect
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.ui.Modifier
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.viewinterop.AndroidView
import androidx.webkit.WebSettingsCompat
import androidx.webkit.WebViewFeature
import ai.openclaw.app.MainViewModel
import java.util.concurrent.atomic.AtomicReference
@SuppressLint("SetJavaScriptEnabled")
@Composable
fun CanvasScreen(viewModel: MainViewModel, visible: Boolean, modifier: Modifier = Modifier) {
val context = LocalContext.current
val isDebuggable = (context.applicationInfo.flags and android.content.pm.ApplicationInfo.FLAG_DEBUGGABLE) != 0
val webViewRef = remember { mutableStateOf<WebView?>(null) }
val currentPageUrlRef = remember { AtomicReference<String?>(null) }
DisposableEffect(viewModel) {
onDispose {
val webView = webViewRef.value ?: return@onDispose
viewModel.canvas.detach(webView)
webView.removeJavascriptInterface(CanvasA2UIActionBridge.interfaceName)
webView.stopLoading()
webView.destroy()
webViewRef.value = null
}
}
AndroidView(
modifier = modifier,
factory = {
WebView(context).apply {
visibility = if (visible) View.VISIBLE else View.INVISIBLE
settings.javaScriptEnabled = true
settings.domStorageEnabled = true
settings.mixedContentMode = WebSettings.MIXED_CONTENT_COMPATIBILITY_MODE
settings.useWideViewPort = false
settings.loadWithOverviewMode = false
settings.builtInZoomControls = false
settings.displayZoomControls = false
settings.setSupportZoom(false)
// targetSdk 33+ ignores Force Dark APIs, so only opt out through the supported
// algorithmic darkening flag when this WebView implementation exposes it.
if (WebViewFeature.isFeatureSupported(WebViewFeature.ALGORITHMIC_DARKENING)) {
WebSettingsCompat.setAlgorithmicDarkeningAllowed(settings, false)
}
if (isDebuggable) {
Log.d("OpenClawWebView", "userAgent: ${settings.userAgentString}")
}
isScrollContainer = true
overScrollMode = View.OVER_SCROLL_IF_CONTENT_SCROLLS
isVerticalScrollBarEnabled = true
isHorizontalScrollBarEnabled = true
webViewClient =
object : WebViewClient() {
override fun onPageStarted(
view: WebView,
url: String?,
favicon: android.graphics.Bitmap?,
) {
currentPageUrlRef.set(url)
}
override fun onReceivedError(
view: WebView,
request: WebResourceRequest,
error: WebResourceError,
) {
if (!isDebuggable || !request.isForMainFrame) return
Log.e("OpenClawWebView", "onReceivedError: ${error.errorCode} ${error.description} ${request.url}")
}
override fun onReceivedHttpError(
view: WebView,
request: WebResourceRequest,
errorResponse: WebResourceResponse,
) {
if (!isDebuggable || !request.isForMainFrame) return
Log.e(
"OpenClawWebView",
"onReceivedHttpError: ${errorResponse.statusCode} ${errorResponse.reasonPhrase} ${request.url}",
)
}
override fun onPageFinished(view: WebView, url: String?) {
currentPageUrlRef.set(url)
if (isDebuggable) {
Log.d("OpenClawWebView", "onPageFinished: $url")
}
viewModel.canvas.onPageFinished()
}
override fun onRenderProcessGone(
view: WebView,
detail: android.webkit.RenderProcessGoneDetail,
): Boolean {
if (isDebuggable) {
Log.e(
"OpenClawWebView",
"onRenderProcessGone didCrash=${detail.didCrash()} priorityAtExit=${detail.rendererPriorityAtExit()}",
)
}
return true
}
}
webChromeClient =
object : WebChromeClient() {
override fun onConsoleMessage(consoleMessage: ConsoleMessage?): Boolean {
if (!isDebuggable) return false
val msg = consoleMessage ?: return false
Log.d(
"OpenClawWebView",
"console ${msg.messageLevel()} @ ${msg.sourceId()}:${msg.lineNumber()} ${msg.message()}",
)
return false
}
}
val bridge =
CanvasA2UIActionBridge(
isTrustedPage = { viewModel.isTrustedCanvasActionUrl(currentPageUrlRef.get()) },
) { payload ->
viewModel.handleCanvasA2UIActionFromWebView(payload)
}
addJavascriptInterface(bridge, CanvasA2UIActionBridge.interfaceName)
viewModel.canvas.attach(this)
webViewRef.value = this
}
},
update = { webView ->
webView.visibility = if (visible) View.VISIBLE else View.INVISIBLE
if (visible) {
webView.resumeTimers()
webView.onResume()
} else {
webView.onPause()
webView.pauseTimers()
}
},
)
}
internal class CanvasA2UIActionBridge(
private val isTrustedPage: () -> Boolean,
private val onMessage: (String) -> Unit,
) {
@JavascriptInterface
fun postMessage(payload: String?) {
val msg = payload?.trim().orEmpty()
if (msg.isEmpty()) return
if (!isTrustedPage()) return
onMessage(msg)
}
companion object {
const val interfaceName: String = "openclawCanvasA2UIAction"
}
}

View file

@ -0,0 +1,10 @@
package ai.openclaw.app.ui
import androidx.compose.runtime.Composable
import ai.openclaw.app.MainViewModel
import ai.openclaw.app.ui.chat.ChatSheetContent
@Composable
fun ChatSheet(viewModel: MainViewModel) {
ChatSheetContent(viewModel = viewModel)
}

View file

@ -0,0 +1,655 @@
package ai.openclaw.app.ui
import androidx.compose.foundation.BorderStroke
import androidx.compose.animation.AnimatedVisibility
import androidx.compose.foundation.background
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.PaddingValues
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.Spacer
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.height
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.layout.width
import androidx.compose.foundation.rememberScrollState
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.foundation.text.KeyboardOptions
import androidx.compose.foundation.verticalScroll
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.filled.Cloud
import androidx.compose.material.icons.filled.ContentCopy
import androidx.compose.material.icons.filled.ExpandLess
import androidx.compose.material.icons.filled.ExpandMore
import androidx.compose.material.icons.filled.Link
import androidx.compose.material.icons.filled.PowerSettingsNew
import androidx.compose.material3.AlertDialog
import androidx.compose.material3.Button
import androidx.compose.material3.ButtonDefaults
import androidx.compose.material3.HorizontalDivider
import androidx.compose.material3.Icon
import androidx.compose.material3.OutlinedTextField
import androidx.compose.material3.OutlinedTextFieldDefaults
import androidx.compose.material3.Surface
import androidx.compose.material3.Switch
import androidx.compose.material3.SwitchDefaults
import androidx.compose.material3.Text
import androidx.compose.material3.TextButton
import androidx.compose.runtime.Composable
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.saveable.rememberSaveable
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.text.font.FontFamily
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.text.input.KeyboardType
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.unit.dp
import ai.openclaw.app.MainViewModel
import ai.openclaw.app.gateway.GatewayEndpoint
import ai.openclaw.app.ui.mobileCardSurface
private enum class ConnectInputMode {
SetupCode,
Manual,
}
@Composable
fun ConnectTabScreen(viewModel: MainViewModel) {
val context = LocalContext.current
val statusText by viewModel.statusText.collectAsState()
val isConnected by viewModel.isConnected.collectAsState()
val remoteAddress by viewModel.remoteAddress.collectAsState()
val manualHost by viewModel.manualHost.collectAsState()
val manualPort by viewModel.manualPort.collectAsState()
val manualTls by viewModel.manualTls.collectAsState()
val manualEnabled by viewModel.manualEnabled.collectAsState()
val gatewayToken by viewModel.gatewayToken.collectAsState()
val gatewayBootstrapToken by viewModel.gatewayBootstrapToken.collectAsState()
val pendingTrust by viewModel.pendingGatewayTrust.collectAsState()
var advancedOpen by rememberSaveable { mutableStateOf(false) }
var inputMode by
remember(manualEnabled, manualHost, gatewayToken) {
mutableStateOf(
if (manualEnabled || manualHost.isNotBlank() || gatewayToken.trim().isNotEmpty()) {
ConnectInputMode.Manual
} else {
ConnectInputMode.SetupCode
},
)
}
var setupCode by rememberSaveable { mutableStateOf("") }
var manualHostInput by rememberSaveable { mutableStateOf(manualHost.ifBlank { "10.0.2.2" }) }
var manualPortInput by rememberSaveable { mutableStateOf(manualPort.toString()) }
var manualTlsInput by rememberSaveable { mutableStateOf(manualTls) }
var passwordInput by rememberSaveable { mutableStateOf("") }
var validationText by rememberSaveable { mutableStateOf<String?>(null) }
if (pendingTrust != null) {
val prompt = pendingTrust!!
AlertDialog(
onDismissRequest = { viewModel.declineGatewayTrustPrompt() },
containerColor = mobileCardSurface,
title = { Text("Trust this gateway?", style = mobileHeadline, color = mobileText) },
text = {
Text(
"First-time TLS connection.\n\nVerify this SHA-256 fingerprint before trusting:\n${prompt.fingerprintSha256}",
style = mobileCallout,
color = mobileText,
)
},
confirmButton = {
TextButton(
onClick = { viewModel.acceptGatewayTrustPrompt() },
colors = ButtonDefaults.textButtonColors(contentColor = mobileAccent),
) {
Text("Trust and continue")
}
},
dismissButton = {
TextButton(
onClick = { viewModel.declineGatewayTrustPrompt() },
colors = ButtonDefaults.textButtonColors(contentColor = mobileTextSecondary),
) {
Text("Cancel")
}
},
)
}
val setupResolvedEndpoint = remember(setupCode) { decodeGatewaySetupCode(setupCode)?.url?.let { parseGatewayEndpoint(it)?.displayUrl } }
val manualResolvedEndpoint = remember(manualHostInput, manualPortInput, manualTlsInput) {
composeGatewayManualUrl(manualHostInput, manualPortInput, manualTlsInput)?.let { parseGatewayEndpoint(it)?.displayUrl }
}
val activeEndpoint =
remember(isConnected, remoteAddress, setupResolvedEndpoint, manualResolvedEndpoint, inputMode) {
when {
isConnected && !remoteAddress.isNullOrBlank() -> remoteAddress!!
inputMode == ConnectInputMode.SetupCode -> setupResolvedEndpoint ?: "Not set"
else -> manualResolvedEndpoint ?: "Not set"
}
}
val showDiagnostics = !isConnected && gatewayStatusHasDiagnostics(statusText)
val pairingRequired = !isConnected && gatewayStatusLooksLikePairing(statusText)
val statusLabel = gatewayStatusForDisplay(statusText)
PairingAutoRetryEffect(enabled = pairingRequired) {
viewModel.refreshGatewayConnection()
}
Column(
modifier = Modifier.verticalScroll(rememberScrollState()).padding(horizontal = 20.dp, vertical = 16.dp),
verticalArrangement = Arrangement.spacedBy(14.dp),
) {
Column(verticalArrangement = Arrangement.spacedBy(6.dp)) {
Text("Gateway Connection", style = mobileTitle1, color = mobileText)
Text(
if (isConnected) "Your gateway is active and ready." else "Connect to your gateway to get started.",
style = mobileCallout,
color = mobileTextSecondary,
)
}
// Status cards in a unified card group
Surface(
modifier = Modifier.fillMaxWidth(),
shape = RoundedCornerShape(14.dp),
color = mobileCardSurface,
border = BorderStroke(1.dp, mobileBorder),
) {
Column {
Row(
modifier = Modifier.fillMaxWidth().padding(horizontal = 14.dp, vertical = 12.dp),
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.spacedBy(12.dp),
) {
Surface(
shape = RoundedCornerShape(10.dp),
color = mobileAccentSoft,
) {
Icon(
imageVector = Icons.Default.Link,
contentDescription = null,
modifier = Modifier.padding(8.dp).size(18.dp),
tint = mobileAccent,
)
}
Column(verticalArrangement = Arrangement.spacedBy(2.dp)) {
Text("Endpoint", style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold), color = mobileTextSecondary)
Text(activeEndpoint, style = mobileBody.copy(fontFamily = FontFamily.Monospace), color = mobileText)
}
}
HorizontalDivider(color = mobileBorder)
Row(
modifier = Modifier.fillMaxWidth().padding(horizontal = 14.dp, vertical = 12.dp),
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.spacedBy(12.dp),
) {
Surface(
shape = RoundedCornerShape(10.dp),
color = if (isConnected) mobileSuccessSoft else mobileSurface,
) {
Icon(
imageVector = Icons.Default.Cloud,
contentDescription = null,
modifier = Modifier.padding(8.dp).size(18.dp),
tint = if (isConnected) mobileSuccess else mobileTextTertiary,
)
}
Column(verticalArrangement = Arrangement.spacedBy(2.dp)) {
Text("Status", style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold), color = mobileTextSecondary)
Text(statusText, style = mobileBody, color = if (isConnected) mobileSuccess else mobileText)
}
}
}
}
if (isConnected) {
// Outlined secondary button when connected — don't scream "danger"
Button(
onClick = {
viewModel.disconnect()
validationText = null
},
modifier = Modifier.fillMaxWidth().height(48.dp),
shape = RoundedCornerShape(14.dp),
colors =
ButtonDefaults.buttonColors(
containerColor = mobileCardSurface,
contentColor = mobileDanger,
),
border = BorderStroke(1.dp, mobileDanger.copy(alpha = 0.4f)),
) {
Icon(Icons.Default.PowerSettingsNew, contentDescription = null, modifier = Modifier.size(18.dp))
Spacer(modifier = Modifier.width(8.dp))
Text("Disconnect", style = mobileHeadline.copy(fontWeight = FontWeight.SemiBold))
}
} else {
Button(
onClick = {
if (statusText.contains("operator offline", ignoreCase = true)) {
validationText = null
viewModel.refreshGatewayConnection()
return@Button
}
val config =
resolveGatewayConnectConfig(
useSetupCode = inputMode == ConnectInputMode.SetupCode,
setupCode = setupCode,
savedManualHost = manualHost,
savedManualPort = manualPort.toString(),
savedManualTls = manualTls,
manualHostInput = manualHostInput,
manualPortInput = manualPortInput,
manualTlsInput = manualTlsInput,
fallbackBootstrapToken = gatewayBootstrapToken,
fallbackToken = gatewayToken,
fallbackPassword = passwordInput,
)
if (config == null) {
validationText =
if (inputMode == ConnectInputMode.SetupCode) {
val parsedSetup = decodeGatewaySetupCode(setupCode)
if (parsedSetup == null) {
"Paste a valid setup code to connect."
} else {
val parsedGateway = parseGatewayEndpointResult(parsedSetup.url)
gatewayEndpointValidationMessage(
parsedGateway.error ?: GatewayEndpointValidationError.INVALID_URL,
GatewayEndpointInputSource.SETUP_CODE,
)
}
} else {
val manualUrl = composeGatewayManualUrl(manualHostInput, manualPortInput, manualTlsInput)
val parsedGateway = manualUrl?.let(::parseGatewayEndpointResult)
gatewayEndpointValidationMessage(
parsedGateway?.error ?: GatewayEndpointValidationError.INVALID_URL,
GatewayEndpointInputSource.MANUAL,
)
}
return@Button
}
validationText = null
if (inputMode == ConnectInputMode.SetupCode) {
viewModel.resetGatewaySetupAuth()
}
viewModel.setManualEnabled(true)
viewModel.setManualHost(config.host)
viewModel.setManualPort(config.port)
viewModel.setManualTls(config.tls)
viewModel.setGatewayBootstrapToken(config.bootstrapToken)
if (config.token.isNotBlank()) {
viewModel.setGatewayToken(config.token)
} else if (config.bootstrapToken.isNotBlank()) {
viewModel.setGatewayToken("")
}
viewModel.setGatewayPassword(config.password)
viewModel.connect(
GatewayEndpoint.manual(host = config.host, port = config.port),
token = config.token.ifEmpty { null },
bootstrapToken = config.bootstrapToken.ifEmpty { null },
password = config.password.ifEmpty { null },
)
},
modifier = Modifier.fillMaxWidth().height(52.dp),
shape = RoundedCornerShape(14.dp),
colors =
ButtonDefaults.buttonColors(
containerColor = mobileAccent,
contentColor = Color.White,
),
) {
Text("Connect Gateway", style = mobileHeadline.copy(fontWeight = FontWeight.Bold))
}
}
if (showDiagnostics) {
Surface(
modifier = Modifier.fillMaxWidth(),
shape = RoundedCornerShape(14.dp),
color = mobileWarningSoft,
border = BorderStroke(1.dp, mobileWarning.copy(alpha = 0.25f)),
) {
Column(
modifier = Modifier.fillMaxWidth().padding(horizontal = 14.dp, vertical = 14.dp),
verticalArrangement = Arrangement.spacedBy(10.dp),
) {
Text(if (pairingRequired) "Pairing required" else "Last gateway error", style = mobileHeadline, color = mobileWarning)
Text(statusLabel, style = mobileBody.copy(fontFamily = FontFamily.Monospace), color = mobileText)
if (pairingRequired) {
Text(
"Approve this phone on the gateway. OpenClaw retries automatically while this screen stays open.",
style = mobileCallout,
color = mobileTextSecondary,
)
CommandBlock("openclaw devices list")
CommandBlock("openclaw devices approve <requestId>")
}
Text("OpenClaw Android ${openClawAndroidVersionLabel()}", style = mobileCaption1, color = mobileTextSecondary)
Button(
onClick = {
copyGatewayDiagnosticsReport(
context = context,
screen = "connect tab",
gatewayAddress = activeEndpoint,
statusText = statusLabel,
)
},
modifier = Modifier.fillMaxWidth().height(46.dp),
shape = RoundedCornerShape(12.dp),
colors =
ButtonDefaults.buttonColors(
containerColor = mobileCardSurface,
contentColor = mobileWarning,
),
border = BorderStroke(1.dp, mobileWarning.copy(alpha = 0.3f)),
) {
Icon(Icons.Default.ContentCopy, contentDescription = null, modifier = Modifier.size(18.dp))
Spacer(modifier = Modifier.width(8.dp))
Text("Copy Report for Claw", style = mobileCallout.copy(fontWeight = FontWeight.Bold))
}
}
}
}
Surface(
modifier = Modifier.fillMaxWidth(),
shape = RoundedCornerShape(14.dp),
color = mobileSurface,
border = BorderStroke(1.dp, mobileBorder),
onClick = { advancedOpen = !advancedOpen },
) {
Row(
modifier = Modifier.fillMaxWidth().padding(horizontal = 14.dp, vertical = 12.dp),
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.SpaceBetween,
) {
Column(verticalArrangement = Arrangement.spacedBy(2.dp)) {
Text("Advanced controls", style = mobileHeadline, color = mobileText)
Text("Setup code, endpoint, TLS, token, password, onboarding.", style = mobileCaption1, color = mobileTextSecondary)
}
Icon(
imageVector = if (advancedOpen) Icons.Default.ExpandLess else Icons.Default.ExpandMore,
contentDescription = if (advancedOpen) "Collapse advanced controls" else "Expand advanced controls",
tint = mobileTextSecondary,
)
}
}
AnimatedVisibility(visible = advancedOpen) {
Surface(
modifier = Modifier.fillMaxWidth(),
shape = RoundedCornerShape(14.dp),
color = mobileCardSurface,
border = BorderStroke(1.dp, mobileBorder),
) {
Column(
modifier = Modifier.fillMaxWidth().padding(horizontal = 14.dp, vertical = 14.dp),
verticalArrangement = Arrangement.spacedBy(12.dp),
) {
Text("Connection method", style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold), color = mobileTextSecondary)
Row(horizontalArrangement = Arrangement.spacedBy(8.dp)) {
MethodChip(
label = "Setup Code",
active = inputMode == ConnectInputMode.SetupCode,
onClick = { inputMode = ConnectInputMode.SetupCode },
)
MethodChip(
label = "Manual",
active = inputMode == ConnectInputMode.Manual,
onClick = { inputMode = ConnectInputMode.Manual },
)
}
Text("Run these on the gateway host:", style = mobileCallout, color = mobileTextSecondary)
CommandBlock("openclaw qr --setup-code-only")
CommandBlock("openclaw qr --json")
Text(
"For Tailscale or public hosts, use wss:// or Tailscale Serve. Private LAN ws:// remains supported.",
style = mobileCaption1,
color = mobileTextSecondary,
)
if (inputMode == ConnectInputMode.SetupCode) {
Text("Setup Code", style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold), color = mobileTextSecondary)
OutlinedTextField(
value = setupCode,
onValueChange = {
setupCode = it
validationText = null
},
placeholder = { Text("Paste setup code", style = mobileBody, color = mobileTextTertiary) },
modifier = Modifier.fillMaxWidth(),
minLines = 3,
maxLines = 5,
keyboardOptions = KeyboardOptions(keyboardType = KeyboardType.Ascii),
textStyle = mobileBody.copy(fontFamily = FontFamily.Monospace, color = mobileText),
shape = RoundedCornerShape(14.dp),
colors = outlinedColors(),
)
if (!setupResolvedEndpoint.isNullOrBlank()) {
EndpointPreview(endpoint = setupResolvedEndpoint)
}
} else {
Row(horizontalArrangement = Arrangement.spacedBy(8.dp)) {
QuickFillChip(
label = "Android Emulator",
onClick = {
manualHostInput = "10.0.2.2"
manualPortInput = "18789"
manualTlsInput = false
validationText = null
},
)
QuickFillChip(
label = "Localhost",
onClick = {
manualHostInput = "127.0.0.1"
manualPortInput = "18789"
manualTlsInput = false
validationText = null
},
)
}
Text("Host", style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold), color = mobileTextSecondary)
OutlinedTextField(
value = manualHostInput,
onValueChange = {
manualHostInput = it
validationText = null
},
placeholder = { Text("10.0.2.2", style = mobileBody, color = mobileTextTertiary) },
modifier = Modifier.fillMaxWidth(),
singleLine = true,
keyboardOptions = KeyboardOptions(keyboardType = KeyboardType.Uri),
textStyle = mobileBody.copy(color = mobileText),
shape = RoundedCornerShape(14.dp),
colors = outlinedColors(),
)
Text(
if (manualTlsInput) "Port (optional, defaults to 443)" else "Port",
style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold),
color = mobileTextSecondary,
)
OutlinedTextField(
value = manualPortInput,
onValueChange = {
manualPortInput = it
validationText = null
},
placeholder = { Text(if (manualTlsInput) "443" else "18789", style = mobileBody, color = mobileTextTertiary) },
modifier = Modifier.fillMaxWidth(),
singleLine = true,
keyboardOptions = KeyboardOptions(keyboardType = KeyboardType.Number),
textStyle = mobileBody.copy(fontFamily = FontFamily.Monospace, color = mobileText),
shape = RoundedCornerShape(14.dp),
colors = outlinedColors(),
)
Row(
modifier = Modifier.fillMaxWidth(),
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.SpaceBetween,
) {
Column(verticalArrangement = Arrangement.spacedBy(2.dp)) {
Text("Use TLS", style = mobileHeadline, color = mobileText)
Text(
"Turn this on for Tailscale or public hosts. Private LAN ws:// remains supported.",
style = mobileCallout,
color = mobileTextSecondary,
)
}
Switch(
checked = manualTlsInput,
onCheckedChange = {
manualTlsInput = it
validationText = null
},
colors =
SwitchDefaults.colors(
checkedTrackColor = mobileAccent,
uncheckedTrackColor = mobileBorderStrong,
checkedThumbColor = Color.White,
uncheckedThumbColor = Color.White,
),
)
}
Text("Token (optional)", style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold), color = mobileTextSecondary)
OutlinedTextField(
value = gatewayToken,
onValueChange = { viewModel.setGatewayToken(it) },
placeholder = { Text("token", style = mobileBody, color = mobileTextTertiary) },
modifier = Modifier.fillMaxWidth(),
singleLine = true,
keyboardOptions = KeyboardOptions(keyboardType = KeyboardType.Ascii),
textStyle = mobileBody.copy(color = mobileText),
shape = RoundedCornerShape(14.dp),
colors = outlinedColors(),
)
Text("Password (optional)", style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold), color = mobileTextSecondary)
OutlinedTextField(
value = passwordInput,
onValueChange = { passwordInput = it },
placeholder = { Text("password", style = mobileBody, color = mobileTextTertiary) },
modifier = Modifier.fillMaxWidth(),
singleLine = true,
keyboardOptions = KeyboardOptions(keyboardType = KeyboardType.Ascii),
textStyle = mobileBody.copy(color = mobileText),
shape = RoundedCornerShape(14.dp),
colors = outlinedColors(),
)
if (!manualResolvedEndpoint.isNullOrBlank()) {
EndpointPreview(endpoint = manualResolvedEndpoint)
}
}
HorizontalDivider(color = mobileBorder)
TextButton(onClick = { viewModel.setOnboardingCompleted(false) }) {
Text("Run onboarding again", style = mobileCallout.copy(fontWeight = FontWeight.SemiBold), color = mobileAccent)
}
}
}
}
if (!validationText.isNullOrBlank()) {
Text(validationText!!, style = mobileCaption1, color = mobileWarning)
}
}
}
@Composable
private fun MethodChip(label: String, active: Boolean, onClick: () -> Unit) {
Button(
onClick = onClick,
modifier = Modifier.height(40.dp),
shape = RoundedCornerShape(12.dp),
contentPadding = PaddingValues(horizontal = 12.dp, vertical = 8.dp),
colors =
ButtonDefaults.buttonColors(
containerColor = if (active) mobileAccent else mobileSurface,
contentColor = if (active) Color.White else mobileText,
),
border = BorderStroke(1.dp, if (active) mobileAccentBorderStrong else mobileBorderStrong),
) {
Text(label, style = mobileCaption1.copy(fontWeight = FontWeight.Bold))
}
}
@Composable
private fun QuickFillChip(label: String, onClick: () -> Unit) {
Button(
onClick = onClick,
shape = RoundedCornerShape(999.dp),
contentPadding = PaddingValues(horizontal = 12.dp, vertical = 6.dp),
colors =
ButtonDefaults.buttonColors(
containerColor = mobileAccentSoft,
contentColor = mobileAccent,
),
elevation = null,
) {
Text(label, style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold))
}
}
@Composable
private fun CommandBlock(command: String) {
Surface(
modifier = Modifier.fillMaxWidth(),
shape = RoundedCornerShape(12.dp),
color = mobileCodeBg,
border = BorderStroke(1.dp, mobileCodeBorder),
) {
Row(modifier = Modifier.fillMaxWidth(), verticalAlignment = Alignment.CenterVertically) {
Box(modifier = Modifier.width(3.dp).height(42.dp).background(mobileCodeAccent))
Text(
text = command,
modifier = Modifier.padding(horizontal = 12.dp, vertical = 10.dp),
style = mobileCallout.copy(fontFamily = FontFamily.Monospace),
color = mobileCodeText,
)
}
}
}
@Composable
private fun EndpointPreview(endpoint: String) {
Column(verticalArrangement = Arrangement.spacedBy(4.dp)) {
HorizontalDivider(color = mobileBorder)
Text("Resolved endpoint", style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold), color = mobileTextSecondary)
Text(endpoint, style = mobileCallout.copy(fontFamily = FontFamily.Monospace), color = mobileText)
HorizontalDivider(color = mobileBorder)
}
}
@Composable
private fun outlinedColors() =
OutlinedTextFieldDefaults.colors(
focusedContainerColor = mobileSurface,
unfocusedContainerColor = mobileSurface,
focusedBorderColor = mobileAccent,
unfocusedBorderColor = mobileBorder,
focusedTextColor = mobileText,
unfocusedTextColor = mobileText,
cursorColor = mobileAccent,
)

View file

@ -0,0 +1,271 @@
package ai.openclaw.app.ui
import ai.openclaw.app.gateway.isPrivateLanGatewayHost
import java.util.Base64
import java.util.Locale
import java.net.URI
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.contentOrNull
import kotlinx.serialization.json.jsonObject
internal data class GatewayEndpointConfig(
val host: String,
val port: Int,
val tls: Boolean,
val displayUrl: String,
)
internal data class GatewaySetupCode(
val url: String,
val bootstrapToken: String?,
val token: String?,
val password: String?,
)
internal data class GatewayConnectConfig(
val host: String,
val port: Int,
val tls: Boolean,
val bootstrapToken: String,
val token: String,
val password: String,
)
internal enum class GatewayEndpointValidationError {
INVALID_URL,
INSECURE_REMOTE_URL,
}
internal enum class GatewayEndpointInputSource {
SETUP_CODE,
MANUAL,
QR_SCAN,
}
internal data class GatewayEndpointParseResult(
val config: GatewayEndpointConfig? = null,
val error: GatewayEndpointValidationError? = null,
)
internal data class GatewayScannedSetupCodeResult(
val setupCode: String? = null,
val error: GatewayEndpointValidationError? = null,
)
private val gatewaySetupJson = Json { ignoreUnknownKeys = true }
private const val remoteGatewaySecurityRule =
"Tailscale and public mobile nodes require wss:// or Tailscale Serve. ws:// is allowed for private LAN, localhost, and the Android emulator."
private const val remoteGatewaySecurityFix =
"Use a private LAN host/address, or enable Tailscale Serve / expose a wss:// gateway URL."
internal fun resolveGatewayConnectConfig(
useSetupCode: Boolean,
setupCode: String,
savedManualHost: String,
savedManualPort: String,
savedManualTls: Boolean,
manualHostInput: String,
manualPortInput: String,
manualTlsInput: Boolean,
fallbackBootstrapToken: String,
fallbackToken: String,
fallbackPassword: String,
): GatewayConnectConfig? {
if (useSetupCode) {
val setup = decodeGatewaySetupCode(setupCode) ?: return null
val parsed = parseGatewayEndpointResult(setup.url).config ?: return null
val setupBootstrapToken = setup.bootstrapToken?.trim().orEmpty()
val sharedToken =
when {
!setup.token.isNullOrBlank() -> setup.token.trim()
setupBootstrapToken.isNotEmpty() -> ""
else -> fallbackToken.trim()
}
val sharedPassword =
when {
!setup.password.isNullOrBlank() -> setup.password.trim()
setupBootstrapToken.isNotEmpty() -> ""
else -> fallbackPassword.trim()
}
return GatewayConnectConfig(
host = parsed.host,
port = parsed.port,
tls = parsed.tls,
bootstrapToken = setupBootstrapToken,
token = sharedToken,
password = sharedPassword,
)
}
val manualUrl = composeGatewayManualUrl(manualHostInput, manualPortInput, manualTlsInput) ?: return null
val parsed = parseGatewayEndpointResult(manualUrl).config ?: return null
val savedManualEndpoint =
composeGatewayManualUrl(savedManualHost, savedManualPort, savedManualTls)
?.let { parseGatewayEndpointResult(it).config }
val preserveBootstrapToken =
savedManualEndpoint != null &&
savedManualEndpoint.host == parsed.host &&
savedManualEndpoint.port == parsed.port &&
savedManualEndpoint.tls == parsed.tls &&
fallbackToken.isBlank() &&
fallbackPassword.isBlank()
return GatewayConnectConfig(
host = parsed.host,
port = parsed.port,
tls = parsed.tls,
bootstrapToken = if (preserveBootstrapToken) fallbackBootstrapToken.trim() else "",
token = fallbackToken.trim(),
password = fallbackPassword.trim(),
)
}
internal fun parseGatewayEndpoint(rawInput: String): GatewayEndpointConfig? {
return parseGatewayEndpointResult(rawInput).config
}
internal fun parseGatewayEndpointResult(rawInput: String): GatewayEndpointParseResult {
val raw = rawInput.trim()
if (raw.isEmpty()) return GatewayEndpointParseResult(error = GatewayEndpointValidationError.INVALID_URL)
val normalized = if (raw.contains("://")) raw else "https://$raw"
val uri =
runCatching { URI(normalized) }.getOrNull()
?: return GatewayEndpointParseResult(error = GatewayEndpointValidationError.INVALID_URL)
val host = uri.host?.trim()?.trim('[', ']').orEmpty()
if (host.isEmpty()) return GatewayEndpointParseResult(error = GatewayEndpointValidationError.INVALID_URL)
val scheme = uri.scheme?.trim()?.lowercase(Locale.US).orEmpty()
val tls =
when (scheme) {
"ws", "http" -> false
"wss", "https" -> true
else -> true
}
if (!tls && !isPrivateLanGatewayHost(host)) {
return GatewayEndpointParseResult(error = GatewayEndpointValidationError.INSECURE_REMOTE_URL)
}
val defaultPort =
when (scheme) {
"wss", "https" -> 443
"ws", "http" -> 18789
else -> 443
}
val displayPort =
when (scheme) {
"wss", "https" -> 443
"ws", "http" -> 80
else -> 443
}
val port = uri.port.takeIf { it in 1..65535 } ?: defaultPort
val displayHost = if (host.contains(":")) "[$host]" else host
val displayUrl =
if (port == displayPort && defaultPort == displayPort) {
"${if (tls) "https" else "http"}://$displayHost"
} else {
"${if (tls) "https" else "http"}://$displayHost:$port"
}
return GatewayEndpointParseResult(
config = GatewayEndpointConfig(host = host, port = port, tls = tls, displayUrl = displayUrl),
)
}
internal fun decodeGatewaySetupCode(rawInput: String): GatewaySetupCode? {
val trimmed = rawInput.trim()
if (trimmed.isEmpty()) return null
val padded =
trimmed
.replace('-', '+')
.replace('_', '/')
.let { normalized ->
val remainder = normalized.length % 4
if (remainder == 0) normalized else normalized + "=".repeat(4 - remainder)
}
return try {
val decoded = String(Base64.getDecoder().decode(padded), Charsets.UTF_8)
val obj = parseJsonObject(decoded) ?: return null
val url = jsonField(obj, "url").orEmpty()
if (url.isEmpty()) return null
val bootstrapToken = jsonField(obj, "bootstrapToken")
val token = jsonField(obj, "token")
val password = jsonField(obj, "password")
GatewaySetupCode(url = url, bootstrapToken = bootstrapToken, token = token, password = password)
} catch (_: IllegalArgumentException) {
null
}
}
internal fun resolveScannedSetupCode(rawInput: String): String? {
return resolveScannedSetupCodeResult(rawInput).setupCode
}
internal fun resolveScannedSetupCodeResult(rawInput: String): GatewayScannedSetupCodeResult {
val setupCode =
resolveSetupCodeCandidate(rawInput)
?: return GatewayScannedSetupCodeResult(error = GatewayEndpointValidationError.INVALID_URL)
val decoded =
decodeGatewaySetupCode(setupCode)
?: return GatewayScannedSetupCodeResult(error = GatewayEndpointValidationError.INVALID_URL)
val parsed = parseGatewayEndpointResult(decoded.url)
if (parsed.config == null) {
return GatewayScannedSetupCodeResult(error = parsed.error)
}
return GatewayScannedSetupCodeResult(setupCode = setupCode)
}
internal fun gatewayEndpointValidationMessage(
error: GatewayEndpointValidationError,
source: GatewayEndpointInputSource,
): String {
return when (error) {
GatewayEndpointValidationError.INSECURE_REMOTE_URL ->
when (source) {
GatewayEndpointInputSource.SETUP_CODE ->
"Setup code points to an insecure remote gateway. $remoteGatewaySecurityRule $remoteGatewaySecurityFix"
GatewayEndpointInputSource.QR_SCAN ->
"QR code points to an insecure remote gateway. $remoteGatewaySecurityRule $remoteGatewaySecurityFix"
GatewayEndpointInputSource.MANUAL ->
"$remoteGatewaySecurityRule $remoteGatewaySecurityFix"
}
GatewayEndpointValidationError.INVALID_URL ->
when (source) {
GatewayEndpointInputSource.SETUP_CODE -> "Setup code has invalid gateway URL."
GatewayEndpointInputSource.QR_SCAN -> "QR code did not contain a valid setup code."
GatewayEndpointInputSource.MANUAL -> "Enter a valid manual endpoint to connect."
}
}
}
internal fun composeGatewayManualUrl(hostInput: String, portInput: String, tls: Boolean): String? {
val host = hostInput.trim()
if (host.isEmpty()) return null
val portTrimmed = portInput.trim()
val port = if (portTrimmed.isEmpty()) {
if (tls) 443 else return null
} else {
portTrimmed.toIntOrNull() ?: return null
}
if (port !in 1..65535) return null
val scheme = if (tls) "https" else "http"
return "$scheme://$host:$port"
}
private fun parseJsonObject(input: String): JsonObject? {
return runCatching { gatewaySetupJson.parseToJsonElement(input).jsonObject }.getOrNull()
}
private fun resolveSetupCodeCandidate(rawInput: String): String? {
val trimmed = rawInput.trim()
if (trimmed.isEmpty()) return null
val qrSetupCode = parseJsonObject(trimmed)?.let { jsonField(it, "setupCode") }
return qrSetupCode ?: trimmed
}
private fun jsonField(obj: JsonObject, key: String): String? {
val value = (obj[key] as? JsonPrimitive)?.contentOrNull?.trim().orEmpty()
return value.ifEmpty { null }
}

View file

@ -0,0 +1,78 @@
package ai.openclaw.app.ui
import android.content.ClipData
import android.content.ClipboardManager
import android.content.Context
import android.os.Build
import android.widget.Toast
import ai.openclaw.app.BuildConfig
internal fun openClawAndroidVersionLabel(): String {
val versionName = BuildConfig.VERSION_NAME.trim().ifEmpty { "dev" }
return if (BuildConfig.DEBUG && !versionName.contains("dev", ignoreCase = true)) {
"$versionName-dev"
} else {
versionName
}
}
internal fun gatewayStatusForDisplay(statusText: String): String {
return statusText.trim().ifEmpty { "Offline" }
}
internal fun gatewayStatusHasDiagnostics(statusText: String): Boolean {
val lower = gatewayStatusForDisplay(statusText).lowercase()
return lower != "offline" && !lower.contains("connecting")
}
internal fun gatewayStatusLooksLikePairing(statusText: String): Boolean {
val lower = gatewayStatusForDisplay(statusText).lowercase()
return lower.contains("pair") || lower.contains("approve")
}
internal fun buildGatewayDiagnosticsReport(
screen: String,
gatewayAddress: String,
statusText: String,
): String {
val device =
listOfNotNull(Build.MANUFACTURER, Build.MODEL)
.joinToString(" ")
.trim()
.ifEmpty { "Android" }
val androidVersion = Build.VERSION.RELEASE?.trim().orEmpty().ifEmpty { Build.VERSION.SDK_INT.toString() }
val endpoint = gatewayAddress.trim().ifEmpty { "unknown" }
val status = gatewayStatusForDisplay(statusText)
return """
Help diagnose this OpenClaw Android gateway connection failure.
Please:
- pick one route only: same machine, same LAN, Tailscale, or public URL
- classify this as pairing/auth, TLS trust, wrong advertised route, wrong address/port, or gateway down
- remember: Tailscale/public mobile routes require wss:// or Tailscale Serve; private LAN ws:// is still allowed
- quote the exact app status/error below
- tell me whether `openclaw devices list` should show a pending pairing request
- if more signal is needed, ask for `openclaw qr --json`, `openclaw devices list`, and `openclaw nodes status`
- give the next exact command or tap
Debug info:
- screen: $screen
- app version: ${openClawAndroidVersionLabel()}
- device: $device
- android: $androidVersion (SDK ${Build.VERSION.SDK_INT})
- gateway address: $endpoint
- status/error: $status
""".trimIndent()
}
internal fun copyGatewayDiagnosticsReport(
context: Context,
screen: String,
gatewayAddress: String,
statusText: String,
) {
val clipboard = context.getSystemService(ClipboardManager::class.java) ?: return
val report = buildGatewayDiagnosticsReport(screen = screen, gatewayAddress = gatewayAddress, statusText = statusText)
clipboard.setPrimaryClip(ClipData.newPlainText("OpenClaw gateway diagnostics", report))
Toast.makeText(context, "Copied gateway diagnostics", Toast.LENGTH_SHORT).show()
}

View file

@ -0,0 +1,45 @@
package ai.openclaw.app.ui
import androidx.compose.runtime.Composable
import androidx.compose.runtime.DisposableEffect
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.setValue
import androidx.lifecycle.Lifecycle
import androidx.lifecycle.LifecycleEventObserver
import androidx.lifecycle.compose.LocalLifecycleOwner
import kotlinx.coroutines.delay
internal const val PAIRING_AUTO_RETRY_MS = 6_000L
@Composable
internal fun PairingAutoRetryEffect(enabled: Boolean, onRetry: () -> Unit) {
val lifecycleOwner = LocalLifecycleOwner.current
var lifecycleStarted by
remember(lifecycleOwner) {
mutableStateOf(lifecycleOwner.lifecycle.currentState.isAtLeast(Lifecycle.State.STARTED))
}
DisposableEffect(lifecycleOwner) {
val observer =
LifecycleEventObserver { _, _ ->
lifecycleStarted = lifecycleOwner.lifecycle.currentState.isAtLeast(Lifecycle.State.STARTED)
}
lifecycleOwner.lifecycle.addObserver(observer)
onDispose {
lifecycleOwner.lifecycle.removeObserver(observer)
}
}
LaunchedEffect(enabled, lifecycleStarted) {
if (!enabled || !lifecycleStarted) {
return@LaunchedEffect
}
while (true) {
delay(PAIRING_AUTO_RETRY_MS)
onRetry()
}
}
}

View file

@ -0,0 +1,232 @@
package ai.openclaw.app.ui
import androidx.compose.runtime.Composable
import androidx.compose.runtime.staticCompositionLocalOf
import androidx.compose.ui.graphics.Brush
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.text.TextStyle
import androidx.compose.ui.text.font.Font
import androidx.compose.ui.text.font.FontFamily
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.unit.sp
import ai.openclaw.app.R
// ---------------------------------------------------------------------------
// MobileColors – semantic color tokens with light + dark variants
// ---------------------------------------------------------------------------
internal data class MobileColors(
val surface: Color,
val surfaceStrong: Color,
val cardSurface: Color,
val border: Color,
val borderStrong: Color,
val text: Color,
val textSecondary: Color,
val textTertiary: Color,
val accent: Color,
val accentSoft: Color,
val accentBorderStrong: Color,
val success: Color,
val successSoft: Color,
val warning: Color,
val warningSoft: Color,
val danger: Color,
val dangerSoft: Color,
val codeBg: Color,
val codeText: Color,
val codeBorder: Color,
val codeAccent: Color,
val chipBorderConnected: Color,
val chipBorderConnecting: Color,
val chipBorderWarning: Color,
val chipBorderError: Color,
)
internal fun lightMobileColors() =
MobileColors(
surface = Color(0xFFF6F7FA),
surfaceStrong = Color(0xFFECEEF3),
cardSurface = Color(0xFFFFFFFF),
border = Color(0xFFE5E7EC),
borderStrong = Color(0xFFD6DAE2),
text = Color(0xFF17181C),
textSecondary = Color(0xFF5D6472),
textTertiary = Color(0xFF99A0AE),
accent = Color(0xFF1D5DD8),
accentSoft = Color(0xFFECF3FF),
accentBorderStrong = Color(0xFF184DAF),
success = Color(0xFF2F8C5A),
successSoft = Color(0xFFEEF9F3),
warning = Color(0xFFC8841A),
warningSoft = Color(0xFFFFF8EC),
danger = Color(0xFFD04B4B),
dangerSoft = Color(0xFFFFF2F2),
codeBg = Color(0xFF15171B),
codeText = Color(0xFFE8EAEE),
codeBorder = Color(0xFF2B2E35),
codeAccent = Color(0xFF3FC97A),
chipBorderConnected = Color(0xFFCFEBD8),
chipBorderConnecting = Color(0xFFD5E2FA),
chipBorderWarning = Color(0xFFEED8B8),
chipBorderError = Color(0xFFF3C8C8),
)
internal fun darkMobileColors() =
MobileColors(
surface = Color(0xFF1A1C20),
surfaceStrong = Color(0xFF24262B),
cardSurface = Color(0xFF1E2024),
border = Color(0xFF2E3038),
borderStrong = Color(0xFF3A3D46),
text = Color(0xFFE4E5EA),
textSecondary = Color(0xFFA0A6B4),
textTertiary = Color(0xFF6B7280),
accent = Color(0xFF6EA8FF),
accentSoft = Color(0xFF1A2A44),
accentBorderStrong = Color(0xFF5B93E8),
success = Color(0xFF5FBB85),
successSoft = Color(0xFF152E22),
warning = Color(0xFFE8A844),
warningSoft = Color(0xFF2E2212),
danger = Color(0xFFE87070),
dangerSoft = Color(0xFF2E1616),
codeBg = Color(0xFF111317),
codeText = Color(0xFFE8EAEE),
codeBorder = Color(0xFF2B2E35),
codeAccent = Color(0xFF3FC97A),
chipBorderConnected = Color(0xFF1E4A30),
chipBorderConnecting = Color(0xFF1E3358),
chipBorderWarning = Color(0xFF3E3018),
chipBorderError = Color(0xFF3E1E1E),
)
internal val LocalMobileColors = staticCompositionLocalOf { lightMobileColors() }
internal object MobileColorsAccessor {
val current: MobileColors
@Composable get() = LocalMobileColors.current
}
// ---------------------------------------------------------------------------
// Backward-compatible top-level accessors (composable getters)
// ---------------------------------------------------------------------------
// These allow existing call sites to keep using `mobileSurface`, `mobileText`, etc.
// without converting every file at once. Each resolves to the themed value.
internal val mobileSurface: Color @Composable get() = LocalMobileColors.current.surface
internal val mobileSurfaceStrong: Color @Composable get() = LocalMobileColors.current.surfaceStrong
internal val mobileCardSurface: Color @Composable get() = LocalMobileColors.current.cardSurface
internal val mobileBorder: Color @Composable get() = LocalMobileColors.current.border
internal val mobileBorderStrong: Color @Composable get() = LocalMobileColors.current.borderStrong
internal val mobileText: Color @Composable get() = LocalMobileColors.current.text
internal val mobileTextSecondary: Color @Composable get() = LocalMobileColors.current.textSecondary
internal val mobileTextTertiary: Color @Composable get() = LocalMobileColors.current.textTertiary
internal val mobileAccent: Color @Composable get() = LocalMobileColors.current.accent
internal val mobileAccentSoft: Color @Composable get() = LocalMobileColors.current.accentSoft
internal val mobileAccentBorderStrong: Color @Composable get() = LocalMobileColors.current.accentBorderStrong
internal val mobileSuccess: Color @Composable get() = LocalMobileColors.current.success
internal val mobileSuccessSoft: Color @Composable get() = LocalMobileColors.current.successSoft
internal val mobileWarning: Color @Composable get() = LocalMobileColors.current.warning
internal val mobileWarningSoft: Color @Composable get() = LocalMobileColors.current.warningSoft
internal val mobileDanger: Color @Composable get() = LocalMobileColors.current.danger
internal val mobileDangerSoft: Color @Composable get() = LocalMobileColors.current.dangerSoft
internal val mobileCodeBg: Color @Composable get() = LocalMobileColors.current.codeBg
internal val mobileCodeText: Color @Composable get() = LocalMobileColors.current.codeText
internal val mobileCodeBorder: Color @Composable get() = LocalMobileColors.current.codeBorder
internal val mobileCodeAccent: Color @Composable get() = LocalMobileColors.current.codeAccent
// Background gradient – light fades white→gray, dark fades near-black→dark-gray
internal val mobileBackgroundGradient: Brush
@Composable get() {
val colors = LocalMobileColors.current
return Brush.verticalGradient(
listOf(
colors.surface,
colors.surfaceStrong,
colors.surfaceStrong,
),
)
}
// ---------------------------------------------------------------------------
// Typography tokens (theme-independent)
// ---------------------------------------------------------------------------
internal val mobileFontFamily =
FontFamily(
Font(resId = R.font.manrope_400_regular, weight = FontWeight.Normal),
Font(resId = R.font.manrope_500_medium, weight = FontWeight.Medium),
Font(resId = R.font.manrope_600_semibold, weight = FontWeight.SemiBold),
Font(resId = R.font.manrope_700_bold, weight = FontWeight.Bold),
)
internal val mobileDisplay =
TextStyle(
fontFamily = mobileFontFamily,
fontWeight = FontWeight.Bold,
fontSize = 34.sp,
lineHeight = 40.sp,
letterSpacing = (-0.8).sp,
)
internal val mobileTitle1 =
TextStyle(
fontFamily = mobileFontFamily,
fontWeight = FontWeight.SemiBold,
fontSize = 24.sp,
lineHeight = 30.sp,
letterSpacing = (-0.5).sp,
)
internal val mobileTitle2 =
TextStyle(
fontFamily = mobileFontFamily,
fontWeight = FontWeight.SemiBold,
fontSize = 20.sp,
lineHeight = 26.sp,
letterSpacing = (-0.3).sp,
)
internal val mobileHeadline =
TextStyle(
fontFamily = mobileFontFamily,
fontWeight = FontWeight.SemiBold,
fontSize = 16.sp,
lineHeight = 22.sp,
letterSpacing = (-0.1).sp,
)
internal val mobileBody =
TextStyle(
fontFamily = mobileFontFamily,
fontWeight = FontWeight.Medium,
fontSize = 15.sp,
lineHeight = 22.sp,
)
internal val mobileCallout =
TextStyle(
fontFamily = mobileFontFamily,
fontWeight = FontWeight.Medium,
fontSize = 14.sp,
lineHeight = 20.sp,
)
internal val mobileCaption1 =
TextStyle(
fontFamily = mobileFontFamily,
fontWeight = FontWeight.Medium,
fontSize = 12.sp,
lineHeight = 16.sp,
letterSpacing = 0.2.sp,
)
internal val mobileCaption2 =
TextStyle(
fontFamily = mobileFontFamily,
fontWeight = FontWeight.Medium,
fontSize = 11.sp,
lineHeight = 14.sp,
letterSpacing = 0.4.sp,
)

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,49 @@
package ai.openclaw.app.ui
import android.app.Activity
import androidx.compose.foundation.isSystemInDarkTheme
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.dynamicDarkColorScheme
import androidx.compose.material3.dynamicLightColorScheme
import androidx.compose.runtime.Composable
import androidx.compose.runtime.CompositionLocalProvider
import androidx.compose.runtime.SideEffect
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.platform.LocalView
import androidx.core.view.WindowCompat
@Composable
fun OpenClawTheme(content: @Composable () -> Unit) {
val context = LocalContext.current
val isDark = isSystemInDarkTheme()
val colorScheme = if (isDark) dynamicDarkColorScheme(context) else dynamicLightColorScheme(context)
val mobileColors = if (isDark) darkMobileColors() else lightMobileColors()
val view = LocalView.current
if (!view.isInEditMode) {
SideEffect {
val window = (view.context as Activity).window
WindowCompat.getInsetsController(window, window.decorView)
.isAppearanceLightStatusBars = !isDark
}
}
CompositionLocalProvider(LocalMobileColors provides mobileColors) {
MaterialTheme(colorScheme = colorScheme, content = content)
}
}
@Composable
fun overlayContainerColor(): Color {
val scheme = MaterialTheme.colorScheme
val isDark = isSystemInDarkTheme()
val base = if (isDark) scheme.surfaceContainerLow else scheme.surfaceContainerHigh
// Light mode: background stays dark (canvas), so clamp overlays away from pure-white glare.
return if (isDark) base else base.copy(alpha = 0.88f)
}
@Composable
fun overlayIconColor(): Color {
return MaterialTheme.colorScheme.onSurfaceVariant
}

View file

@ -0,0 +1,350 @@
package ai.openclaw.app.ui
import androidx.compose.foundation.background
import androidx.compose.foundation.BorderStroke
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.WindowInsets
import androidx.compose.foundation.layout.WindowInsetsSides
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.heightIn
import androidx.compose.foundation.layout.ime
import androidx.compose.foundation.layout.navigationBars
import androidx.compose.foundation.layout.only
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.safeDrawing
import androidx.compose.foundation.layout.windowInsetsPadding
import androidx.compose.foundation.layout.consumeWindowInsets
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.automirrored.filled.ScreenShare
import androidx.compose.material.icons.filled.ChatBubble
import androidx.compose.material.icons.filled.CheckCircle
import androidx.compose.material.icons.filled.RecordVoiceOver
import androidx.compose.material.icons.filled.Settings
import androidx.compose.material3.Icon
import androidx.compose.material3.Scaffold
import androidx.compose.material3.Surface
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.saveable.rememberSaveable
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.draw.alpha
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.zIndex
import androidx.compose.ui.graphics.vector.ImageVector
import androidx.compose.ui.platform.LocalDensity
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.unit.dp
import ai.openclaw.app.HomeDestination
import ai.openclaw.app.MainViewModel
private enum class HomeTab(
val label: String,
val icon: ImageVector,
) {
Connect(label = "Connect", icon = Icons.Default.CheckCircle),
Chat(label = "Chat", icon = Icons.Default.ChatBubble),
Voice(label = "Voice", icon = Icons.Default.RecordVoiceOver),
Screen(label = "Screen", icon = Icons.AutoMirrored.Filled.ScreenShare),
Settings(label = "Settings", icon = Icons.Default.Settings),
}
private enum class StatusVisual {
Connected,
Connecting,
Warning,
Error,
Offline,
}
@Composable
fun PostOnboardingTabs(viewModel: MainViewModel, modifier: Modifier = Modifier) {
var activeTab by rememberSaveable { mutableStateOf(HomeTab.Connect) }
var chatTabStarted by rememberSaveable { mutableStateOf(false) }
var screenTabStarted by rememberSaveable { mutableStateOf(false) }
val requestedHomeDestination by viewModel.requestedHomeDestination.collectAsState()
LaunchedEffect(requestedHomeDestination) {
val destination = requestedHomeDestination ?: return@LaunchedEffect
activeTab =
when (destination) {
HomeDestination.Connect -> HomeTab.Connect
HomeDestination.Chat -> HomeTab.Chat
HomeDestination.Voice -> HomeTab.Voice
HomeDestination.Screen -> HomeTab.Screen
HomeDestination.Settings -> HomeTab.Settings
}
viewModel.clearRequestedHomeDestination()
}
// Stop TTS when user navigates away from voice tab, and lazily keep the Chat/Screen tabs
// alive after the first visit so repeated tab switches do not rebuild their UI trees.
LaunchedEffect(activeTab) {
viewModel.setVoiceScreenActive(activeTab == HomeTab.Voice)
if (activeTab == HomeTab.Chat) {
chatTabStarted = true
}
if (activeTab == HomeTab.Screen) {
screenTabStarted = true
}
}
val statusText by viewModel.statusText.collectAsState()
val isConnected by viewModel.isConnected.collectAsState()
val statusVisual =
remember(statusText, isConnected) {
val lower = statusText.lowercase()
when {
isConnected -> StatusVisual.Connected
lower.contains("connecting") || lower.contains("reconnecting") -> StatusVisual.Connecting
lower.contains("pairing") || lower.contains("approval") || lower.contains("auth") -> StatusVisual.Warning
lower.contains("error") || lower.contains("failed") -> StatusVisual.Error
else -> StatusVisual.Offline
}
}
val density = LocalDensity.current
val imeVisible = WindowInsets.ime.getBottom(density) > 0
val hideBottomTabBar = activeTab == HomeTab.Chat && imeVisible
Scaffold(
modifier = modifier,
containerColor = Color.Transparent,
contentWindowInsets = WindowInsets(0, 0, 0, 0),
topBar = {
TopStatusBar(
statusText = statusText,
statusVisual = statusVisual,
)
},
bottomBar = {
if (!hideBottomTabBar) {
BottomTabBar(
activeTab = activeTab,
onSelect = { activeTab = it },
)
}
},
) { innerPadding ->
Box(
modifier =
Modifier
.fillMaxSize()
.padding(innerPadding)
.consumeWindowInsets(innerPadding)
.background(mobileBackgroundGradient),
) {
if (chatTabStarted) {
Box(
modifier =
Modifier
.matchParentSize()
.alpha(if (activeTab == HomeTab.Chat) 1f else 0f)
.zIndex(if (activeTab == HomeTab.Chat) 1f else 0f),
) {
ChatSheet(viewModel = viewModel)
}
}
if (screenTabStarted) {
ScreenTabScreen(
viewModel = viewModel,
visible = activeTab == HomeTab.Screen,
modifier =
Modifier
.matchParentSize()
.alpha(if (activeTab == HomeTab.Screen) 1f else 0f)
.zIndex(if (activeTab == HomeTab.Screen) 1f else 0f),
)
}
when (activeTab) {
HomeTab.Connect -> ConnectTabScreen(viewModel = viewModel)
HomeTab.Chat -> if (!chatTabStarted) ChatSheet(viewModel = viewModel)
HomeTab.Voice -> VoiceTabScreen(viewModel = viewModel)
HomeTab.Screen -> Unit
HomeTab.Settings -> SettingsSheet(viewModel = viewModel)
}
}
}
}
@Composable
private fun ScreenTabScreen(viewModel: MainViewModel, visible: Boolean, modifier: Modifier = Modifier) {
val isConnected by viewModel.isConnected.collectAsState()
var refreshedForCurrentConnection by rememberSaveable(isConnected) { mutableStateOf(false) }
LaunchedEffect(isConnected, visible, refreshedForCurrentConnection) {
if (visible && isConnected && !refreshedForCurrentConnection) {
viewModel.refreshHomeCanvasOverviewIfConnected()
refreshedForCurrentConnection = true
}
}
Box(modifier = modifier.fillMaxSize()) {
CanvasScreen(viewModel = viewModel, visible = visible, modifier = Modifier.fillMaxSize())
}
}
@Composable
private fun TopStatusBar(
statusText: String,
statusVisual: StatusVisual,
) {
val safeInsets = WindowInsets.safeDrawing.only(WindowInsetsSides.Top + WindowInsetsSides.Horizontal)
val (chipBg, chipDot, chipText, chipBorder) =
when (statusVisual) {
StatusVisual.Connected ->
listOf(
mobileSuccessSoft,
mobileSuccess,
mobileSuccess,
LocalMobileColors.current.chipBorderConnected,
)
StatusVisual.Connecting ->
listOf(
mobileAccentSoft,
mobileAccent,
mobileAccent,
LocalMobileColors.current.chipBorderConnecting,
)
StatusVisual.Warning ->
listOf(
mobileWarningSoft,
mobileWarning,
mobileWarning,
LocalMobileColors.current.chipBorderWarning,
)
StatusVisual.Error ->
listOf(
mobileDangerSoft,
mobileDanger,
mobileDanger,
LocalMobileColors.current.chipBorderError,
)
StatusVisual.Offline ->
listOf(
mobileSurface,
mobileTextTertiary,
mobileTextSecondary,
mobileBorder,
)
}
Surface(
modifier = Modifier.fillMaxWidth().windowInsetsPadding(safeInsets),
color = Color.Transparent,
shadowElevation = 0.dp,
) {
Row(
modifier = Modifier.fillMaxWidth().padding(horizontal = 18.dp, vertical = 12.dp),
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.SpaceBetween,
) {
Text(
text = "OpenClaw",
style = mobileTitle2,
color = mobileText,
)
Surface(
shape = RoundedCornerShape(999.dp),
color = chipBg,
border = androidx.compose.foundation.BorderStroke(1.dp, chipBorder),
) {
Row(
modifier = Modifier.padding(horizontal = 10.dp, vertical = 5.dp),
horizontalArrangement = Arrangement.spacedBy(6.dp),
verticalAlignment = Alignment.CenterVertically,
) {
Surface(
modifier = Modifier.padding(top = 1.dp),
color = chipDot,
shape = RoundedCornerShape(999.dp),
) {
Box(modifier = Modifier.padding(4.dp))
}
Text(
text = statusText.trim().ifEmpty { "Offline" },
style = mobileCaption1,
color = chipText,
maxLines = 1,
)
}
}
}
}
}
@Composable
private fun BottomTabBar(
activeTab: HomeTab,
onSelect: (HomeTab) -> Unit,
) {
val safeInsets = WindowInsets.navigationBars.only(WindowInsetsSides.Bottom + WindowInsetsSides.Horizontal)
Box(
modifier =
Modifier
.fillMaxWidth(),
) {
Surface(
modifier = Modifier.fillMaxWidth(),
color = mobileCardSurface.copy(alpha = 0.97f),
shape = RoundedCornerShape(topStart = 24.dp, topEnd = 24.dp),
border = BorderStroke(1.dp, mobileBorder),
shadowElevation = 6.dp,
) {
Row(
modifier =
Modifier
.fillMaxWidth()
.windowInsetsPadding(safeInsets)
.padding(horizontal = 10.dp, vertical = 10.dp),
horizontalArrangement = Arrangement.spacedBy(6.dp),
verticalAlignment = Alignment.CenterVertically,
) {
HomeTab.entries.forEach { tab ->
val active = tab == activeTab
Surface(
onClick = { onSelect(tab) },
modifier = Modifier.weight(1f).heightIn(min = 58.dp),
shape = RoundedCornerShape(16.dp),
color = if (active) mobileAccentSoft else Color.Transparent,
border = if (active) BorderStroke(1.dp, LocalMobileColors.current.chipBorderConnecting) else null,
shadowElevation = 0.dp,
) {
Column(
modifier = Modifier.fillMaxWidth().padding(horizontal = 6.dp, vertical = 7.dp),
horizontalAlignment = Alignment.CenterHorizontally,
verticalArrangement = Arrangement.spacedBy(2.dp),
) {
Icon(
imageVector = tab.icon,
contentDescription = tab.label,
tint = if (active) mobileAccent else mobileTextTertiary,
)
Text(
text = tab.label,
color = if (active) mobileAccent else mobileTextSecondary,
style = mobileCaption2.copy(fontWeight = if (active) FontWeight.Bold else FontWeight.Medium),
)
}
}
}
}
}
}
}

View file

@ -0,0 +1,20 @@
package ai.openclaw.app.ui
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.runtime.Composable
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue
import androidx.compose.ui.Modifier
import ai.openclaw.app.MainViewModel
@Composable
fun RootScreen(viewModel: MainViewModel) {
val onboardingCompleted by viewModel.onboardingCompleted.collectAsState()
if (!onboardingCompleted) {
OnboardingFlow(viewModel = viewModel, modifier = Modifier.fillMaxSize())
return
}
PostOnboardingTabs(viewModel = viewModel, modifier = Modifier.fillMaxSize())
}

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,134 @@
package ai.openclaw.app.ui
import androidx.compose.animation.core.LinearEasing
import androidx.compose.animation.core.RepeatMode
import androidx.compose.animation.core.animateFloat
import androidx.compose.animation.core.infiniteRepeatable
import androidx.compose.animation.core.rememberInfiniteTransition
import androidx.compose.animation.core.tween
import androidx.compose.foundation.Canvas
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.shape.CircleShape
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Surface
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.getValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Brush
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.graphics.drawscope.Stroke
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.unit.dp
@Composable
fun TalkOrbOverlay(
seamColor: Color,
statusText: String,
isListening: Boolean,
isSpeaking: Boolean,
modifier: Modifier = Modifier,
) {
val transition = rememberInfiniteTransition(label = "talk-orb")
val t by
transition.animateFloat(
initialValue = 0f,
targetValue = 1f,
animationSpec =
infiniteRepeatable(
animation = tween(durationMillis = 1500, easing = LinearEasing),
repeatMode = RepeatMode.Restart,
),
label = "pulse",
)
val trimmed = statusText.trim()
val showStatus = trimmed.isNotEmpty() && trimmed != "Off"
val phase =
when {
isSpeaking -> "Speaking"
isListening -> "Listening"
else -> "Thinking"
}
Column(
modifier = modifier.padding(24.dp),
horizontalAlignment = Alignment.CenterHorizontally,
verticalArrangement = Arrangement.spacedBy(12.dp),
) {
Box(contentAlignment = Alignment.Center) {
Canvas(modifier = Modifier.size(360.dp)) {
val center = this.center
val baseRadius = size.minDimension * 0.30f
val ring1 = 1.05f + (t * 0.25f)
val ring2 = 1.20f + (t * 0.55f)
val ringAlpha1 = (1f - t) * 0.34f
val ringAlpha2 = (1f - t) * 0.22f
drawCircle(
color = seamColor.copy(alpha = ringAlpha1),
radius = baseRadius * ring1,
center = center,
style = Stroke(width = 3.dp.toPx()),
)
drawCircle(
color = seamColor.copy(alpha = ringAlpha2),
radius = baseRadius * ring2,
center = center,
style = Stroke(width = 3.dp.toPx()),
)
drawCircle(
brush =
Brush.radialGradient(
colors =
listOf(
seamColor.copy(alpha = 0.92f),
seamColor.copy(alpha = 0.40f),
Color.Black.copy(alpha = 0.56f),
),
center = center,
radius = baseRadius * 1.35f,
),
radius = baseRadius,
center = center,
)
drawCircle(
color = seamColor.copy(alpha = 0.34f),
radius = baseRadius,
center = center,
style = Stroke(width = 1.dp.toPx()),
)
}
}
if (showStatus) {
Surface(
color = Color.Black.copy(alpha = 0.40f),
shape = CircleShape,
) {
Text(
text = trimmed,
modifier = Modifier.padding(horizontal = 14.dp, vertical = 8.dp),
color = Color.White.copy(alpha = 0.92f),
style = MaterialTheme.typography.labelLarge,
fontWeight = FontWeight.SemiBold,
)
}
} else {
Text(
text = phase,
color = Color.White.copy(alpha = 0.80f),
style = MaterialTheme.typography.labelLarge,
fontWeight = FontWeight.SemiBold,
)
}
}
}

View file

@ -0,0 +1,448 @@
package ai.openclaw.app.ui
import android.Manifest
import android.app.Activity
import android.content.Context
import android.content.ContextWrapper
import android.content.Intent
import android.content.pm.PackageManager
import android.net.Uri
import android.provider.Settings
import androidx.activity.compose.rememberLauncherForActivityResult
import androidx.activity.result.contract.ActivityResultContracts
import androidx.compose.foundation.BorderStroke
import androidx.compose.foundation.background
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.PaddingValues
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.Spacer
import androidx.compose.foundation.layout.WindowInsets
import androidx.compose.foundation.layout.WindowInsetsSides
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.height
import androidx.compose.foundation.layout.imePadding
import androidx.compose.foundation.layout.only
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.safeDrawing
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.layout.windowInsetsPadding
import androidx.compose.foundation.lazy.LazyColumn
import androidx.compose.foundation.lazy.items
import androidx.compose.foundation.lazy.rememberLazyListState
import androidx.compose.foundation.shape.CircleShape
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.filled.Mic
import androidx.compose.material.icons.filled.MicOff
import androidx.compose.material.icons.automirrored.filled.VolumeOff
import androidx.compose.material.icons.automirrored.filled.VolumeUp
import androidx.compose.material3.Button
import androidx.compose.material3.ButtonDefaults
import androidx.compose.material3.Icon
import androidx.compose.material3.IconButton
import androidx.compose.material3.IconButtonDefaults
import androidx.compose.material3.Surface
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.DisposableEffect
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.draw.alpha
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.text.style.TextAlign
import androidx.compose.ui.unit.dp
import androidx.compose.ui.unit.sp
import androidx.core.app.ActivityCompat
import androidx.core.content.ContextCompat
import androidx.lifecycle.Lifecycle
import androidx.lifecycle.LifecycleEventObserver
import androidx.lifecycle.compose.LocalLifecycleOwner
import ai.openclaw.app.MainViewModel
import ai.openclaw.app.voice.VoiceConversationEntry
import ai.openclaw.app.voice.VoiceConversationRole
import kotlin.math.max
@Composable
fun VoiceTabScreen(viewModel: MainViewModel) {
val context = LocalContext.current
val lifecycleOwner = LocalLifecycleOwner.current
val activity = remember(context) { context.findActivity() }
val listState = rememberLazyListState()
val gatewayStatus by viewModel.statusText.collectAsState()
val micEnabled by viewModel.micEnabled.collectAsState()
val micCooldown by viewModel.micCooldown.collectAsState()
val speakerEnabled by viewModel.speakerEnabled.collectAsState()
val micStatusText by viewModel.micStatusText.collectAsState()
val micLiveTranscript by viewModel.micLiveTranscript.collectAsState()
val micQueuedMessages by viewModel.micQueuedMessages.collectAsState()
val micConversation by viewModel.micConversation.collectAsState()
val micInputLevel by viewModel.micInputLevel.collectAsState()
val micIsSending by viewModel.micIsSending.collectAsState()
val hasStreamingAssistant = micConversation.any { it.role == VoiceConversationRole.Assistant && it.isStreaming }
val showThinkingBubble = micIsSending && !hasStreamingAssistant
var hasMicPermission by remember { mutableStateOf(context.hasRecordAudioPermission()) }
var pendingMicEnable by remember { mutableStateOf(false) }
DisposableEffect(lifecycleOwner, context) {
val observer =
LifecycleEventObserver { _, event ->
if (event == Lifecycle.Event.ON_RESUME) {
hasMicPermission = context.hasRecordAudioPermission()
}
}
lifecycleOwner.lifecycle.addObserver(observer)
onDispose {
lifecycleOwner.lifecycle.removeObserver(observer)
// Stop TTS when leaving the voice screen
viewModel.setVoiceScreenActive(false)
}
}
val requestMicPermission =
rememberLauncherForActivityResult(ActivityResultContracts.RequestPermission()) { granted ->
hasMicPermission = granted
if (granted && pendingMicEnable) {
viewModel.setMicEnabled(true)
}
pendingMicEnable = false
}
LaunchedEffect(micConversation.size, showThinkingBubble) {
val total = micConversation.size + if (showThinkingBubble) 1 else 0
if (total > 0) {
listState.animateScrollToItem(total - 1)
}
}
Column(
modifier =
Modifier
.fillMaxSize()
.background(mobileBackgroundGradient)
.imePadding()
.windowInsetsPadding(WindowInsets.safeDrawing.only(WindowInsetsSides.Bottom))
.padding(horizontal = 20.dp, vertical = 14.dp),
verticalArrangement = Arrangement.spacedBy(10.dp),
) {
LazyColumn(
state = listState,
modifier = Modifier.fillMaxWidth().weight(1f),
contentPadding = PaddingValues(vertical = 4.dp),
verticalArrangement = Arrangement.spacedBy(10.dp),
) {
if (micConversation.isEmpty() && !showThinkingBubble) {
item {
Box(
modifier = Modifier.fillParentMaxHeight().fillMaxWidth(),
contentAlignment = Alignment.Center,
) {
Column(
horizontalAlignment = Alignment.CenterHorizontally,
verticalArrangement = Arrangement.spacedBy(10.dp),
) {
Icon(
imageVector = Icons.Default.Mic,
contentDescription = null,
modifier = Modifier.size(48.dp),
tint = mobileTextTertiary,
)
Text(
"Tap the mic to start",
style = mobileHeadline,
color = mobileTextSecondary,
)
Text(
"Each pause sends a turn automatically.",
style = mobileCallout,
color = mobileTextTertiary,
)
}
}
}
}
items(items = micConversation, key = { it.id }) { entry ->
VoiceTurnBubble(entry = entry)
}
if (showThinkingBubble) {
item {
VoiceThinkingBubble()
}
}
}
Column(
modifier = Modifier.fillMaxWidth(),
horizontalAlignment = Alignment.CenterHorizontally,
verticalArrangement = Arrangement.spacedBy(6.dp),
) {
if (!micLiveTranscript.isNullOrBlank()) {
Surface(
modifier = Modifier.fillMaxWidth(),
shape = RoundedCornerShape(14.dp),
color = mobileAccentSoft,
border = BorderStroke(1.dp, mobileAccent.copy(alpha = 0.2f)),
) {
Text(
micLiveTranscript!!.trim(),
modifier = Modifier.padding(horizontal = 12.dp, vertical = 10.dp),
style = mobileCallout,
color = mobileText,
)
}
}
// Mic button with input-reactive ring + speaker toggle
Row(
modifier = Modifier.fillMaxWidth(),
horizontalArrangement = Arrangement.Center,
verticalAlignment = Alignment.CenterVertically,
) {
// Speaker toggle
Column(horizontalAlignment = Alignment.CenterHorizontally, verticalArrangement = Arrangement.spacedBy(4.dp)) {
IconButton(
onClick = { viewModel.setSpeakerEnabled(!speakerEnabled) },
modifier = Modifier.size(48.dp),
colors =
IconButtonDefaults.iconButtonColors(
containerColor = if (speakerEnabled) mobileSurface else mobileDangerSoft,
),
) {
Icon(
imageVector = if (speakerEnabled) Icons.AutoMirrored.Filled.VolumeUp else Icons.AutoMirrored.Filled.VolumeOff,
contentDescription = if (speakerEnabled) "Mute speaker" else "Unmute speaker",
modifier = Modifier.size(22.dp),
tint = if (speakerEnabled) mobileTextSecondary else mobileDanger,
)
}
Text(
if (speakerEnabled) "Speaker" else "Muted",
style = mobileCaption2,
color = if (speakerEnabled) mobileTextTertiary else mobileDanger,
)
}
// Ring size = 68dp base + up to 22dp driven by mic input level.
// The outer Box is fixed at 90dp (max ring size) so the ring never shifts the button.
Box(
modifier = Modifier.padding(horizontal = 16.dp).size(90.dp),
contentAlignment = Alignment.Center,
) {
if (micEnabled) {
val ringLevel = micInputLevel.coerceIn(0f, 1f)
val ringSize = 68.dp + (22.dp * max(ringLevel, 0.05f))
Box(
modifier =
Modifier
.size(ringSize)
.background(mobileAccent.copy(alpha = 0.12f + 0.14f * ringLevel), CircleShape),
)
}
Button(
onClick = {
if (micCooldown) return@Button
if (micEnabled) {
viewModel.setMicEnabled(false)
return@Button
}
if (hasMicPermission) {
viewModel.setMicEnabled(true)
} else {
pendingMicEnable = true
requestMicPermission.launch(Manifest.permission.RECORD_AUDIO)
}
},
enabled = !micCooldown,
shape = CircleShape,
contentPadding = PaddingValues(0.dp),
modifier = Modifier.size(60.dp),
colors =
ButtonDefaults.buttonColors(
containerColor = if (micCooldown) mobileTextSecondary else if (micEnabled) mobileDanger else mobileAccent,
contentColor = Color.White,
disabledContainerColor = mobileTextSecondary,
disabledContentColor = Color.White.copy(alpha = 0.5f),
),
) {
Icon(
imageVector = if (micEnabled) Icons.Default.MicOff else Icons.Default.Mic,
contentDescription = if (micEnabled) "Turn microphone off" else "Turn microphone on",
modifier = Modifier.size(24.dp),
)
}
}
// Invisible spacer to balance the row (matches speaker column width)
Column(horizontalAlignment = Alignment.CenterHorizontally) {
Box(modifier = Modifier.size(48.dp))
Spacer(modifier = Modifier.height(4.dp))
Text("", style = mobileCaption2)
}
}
// Status + labels
val queueCount = micQueuedMessages.size
val stateText =
when {
queueCount > 0 -> "$queueCount queued"
micIsSending -> "Sending"
micCooldown -> "Cooldown"
micEnabled -> "Listening"
else -> "Mic off"
}
val stateColor =
when {
micEnabled -> mobileSuccess
micIsSending -> mobileAccent
else -> mobileTextSecondary
}
Surface(
shape = RoundedCornerShape(999.dp),
color = if (micEnabled) mobileSuccessSoft else mobileSurface,
border = BorderStroke(1.dp, if (micEnabled) mobileSuccess.copy(alpha = 0.3f) else mobileBorder),
) {
Text(
"$gatewayStatus · $stateText",
style = mobileCallout.copy(fontWeight = FontWeight.SemiBold),
color = stateColor,
modifier = Modifier.padding(horizontal = 14.dp, vertical = 6.dp),
)
}
if (!hasMicPermission) {
val showRationale =
if (activity == null) {
false
} else {
ActivityCompat.shouldShowRequestPermissionRationale(activity, Manifest.permission.RECORD_AUDIO)
}
Text(
if (showRationale) {
"Microphone permission is required for voice mode."
} else {
"Microphone blocked. Open app settings to enable it."
},
style = mobileCaption1,
color = mobileWarning,
textAlign = TextAlign.Center,
)
Button(
onClick = { openAppSettings(context) },
shape = RoundedCornerShape(12.dp),
colors = ButtonDefaults.buttonColors(containerColor = mobileSurfaceStrong, contentColor = mobileText),
) {
Text("Open settings", style = mobileCallout.copy(fontWeight = FontWeight.SemiBold))
}
}
}
}
}
@Composable
private fun VoiceTurnBubble(entry: VoiceConversationEntry) {
val isUser = entry.role == VoiceConversationRole.User
Row(
modifier = Modifier.fillMaxWidth(),
horizontalArrangement = if (isUser) Arrangement.End else Arrangement.Start,
) {
Surface(
modifier = Modifier.fillMaxWidth(0.90f),
shape = RoundedCornerShape(12.dp),
color = if (isUser) mobileAccentSoft else mobileCardSurface,
border = BorderStroke(1.dp, if (isUser) mobileAccent else mobileBorderStrong),
) {
Column(
modifier = Modifier.fillMaxWidth().padding(horizontal = 11.dp, vertical = 8.dp),
verticalArrangement = Arrangement.spacedBy(3.dp),
) {
Text(
if (isUser) "You" else "OpenClaw",
style = mobileCaption2.copy(fontWeight = FontWeight.SemiBold, letterSpacing = 0.6.sp),
color = if (isUser) mobileAccent else mobileTextSecondary,
)
Text(
if (entry.isStreaming && entry.text.isBlank()) "Listening response…" else entry.text,
style = mobileCallout,
color = mobileText,
)
}
}
}
}
@Composable
private fun VoiceThinkingBubble() {
Row(modifier = Modifier.fillMaxWidth(), horizontalArrangement = Arrangement.Start) {
Surface(
modifier = Modifier.fillMaxWidth(0.68f),
shape = RoundedCornerShape(12.dp),
color = mobileCardSurface,
border = BorderStroke(1.dp, mobileBorderStrong),
) {
Row(
modifier = Modifier.padding(horizontal = 11.dp, vertical = 8.dp),
horizontalArrangement = Arrangement.spacedBy(8.dp),
verticalAlignment = Alignment.CenterVertically,
) {
ThinkingDots(color = mobileTextSecondary)
Text("OpenClaw is thinking…", style = mobileCallout, color = mobileTextSecondary)
}
}
}
}
@Composable
private fun ThinkingDots(color: Color) {
Row(horizontalArrangement = Arrangement.spacedBy(5.dp), verticalAlignment = Alignment.CenterVertically) {
ThinkingDot(alpha = 0.38f, color = color)
ThinkingDot(alpha = 0.62f, color = color)
ThinkingDot(alpha = 0.90f, color = color)
}
}
@Composable
private fun ThinkingDot(alpha: Float, color: Color) {
Surface(
modifier = Modifier.size(6.dp).alpha(alpha),
shape = CircleShape,
color = color,
) {}
}
private fun Context.hasRecordAudioPermission(): Boolean {
return (
ContextCompat.checkSelfPermission(this, Manifest.permission.RECORD_AUDIO) ==
PackageManager.PERMISSION_GRANTED
)
}
private fun Context.findActivity(): Activity? =
when (this) {
is Activity -> this
is ContextWrapper -> baseContext.findActivity()
else -> null
}
private fun openAppSettings(context: Context) {
val intent =
Intent(
Settings.ACTION_APPLICATION_DETAILS_SETTINGS,
Uri.fromParts("package", context.packageName, null),
)
context.startActivity(intent)
}

View file

@ -0,0 +1,39 @@
package ai.openclaw.app.ui.chat
import androidx.compose.runtime.Composable
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.setValue
import androidx.compose.ui.graphics.ImageBitmap
import androidx.compose.ui.graphics.asImageBitmap
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
internal data class Base64ImageState(
val image: ImageBitmap?,
val failed: Boolean,
)
@Composable
internal fun rememberBase64ImageState(base64: String): Base64ImageState {
var image by remember(base64) { mutableStateOf<ImageBitmap?>(null) }
var failed by remember(base64) { mutableStateOf(false) }
LaunchedEffect(base64) {
failed = false
image =
withContext(Dispatchers.Default) {
try {
val bitmap = decodeBase64Bitmap(base64) ?: return@withContext null
bitmap.asImageBitmap()
} catch (_: Throwable) {
null
}
}
if (image == null) failed = true
}
return Base64ImageState(image = image, failed = failed)
}

View file

@ -0,0 +1,393 @@
package ai.openclaw.app.ui.chat
import androidx.compose.foundation.BorderStroke
import androidx.compose.foundation.horizontalScroll
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.PaddingValues
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.Spacer
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.height
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.layout.width
import androidx.compose.foundation.rememberScrollState
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.automirrored.filled.Send
import androidx.compose.material.icons.filled.ArrowDropDown
import androidx.compose.material.icons.filled.AttachFile
import androidx.compose.material.icons.filled.Refresh
import androidx.compose.material.icons.filled.Stop
import androidx.compose.material3.Button
import androidx.compose.material3.ButtonDefaults
import androidx.compose.material3.CircularProgressIndicator
import androidx.compose.material3.DropdownMenu
import androidx.compose.material3.DropdownMenuItem
import androidx.compose.material3.Icon
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.OutlinedTextField
import androidx.compose.material3.OutlinedTextFieldDefaults
import androidx.compose.material3.Surface
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.saveable.rememberSaveable
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.text.style.TextOverflow
import androidx.compose.ui.unit.dp
import androidx.compose.ui.unit.sp
import ai.openclaw.app.ui.mobileAccent
import ai.openclaw.app.ui.mobileAccentBorderStrong
import ai.openclaw.app.ui.mobileAccentSoft
import ai.openclaw.app.ui.mobileBorder
import ai.openclaw.app.ui.mobileBorderStrong
import ai.openclaw.app.ui.mobileCallout
import ai.openclaw.app.ui.mobileCaption1
import ai.openclaw.app.ui.mobileCardSurface
import ai.openclaw.app.ui.mobileHeadline
import ai.openclaw.app.ui.mobileSurface
import ai.openclaw.app.ui.mobileText
import ai.openclaw.app.ui.mobileTextSecondary
import ai.openclaw.app.ui.mobileTextTertiary
internal data class DraftApplication(
val input: String,
val lastAppliedDraft: String?,
val consumed: Boolean,
)
internal fun applyDraftText(
draftText: String?,
currentInput: String,
lastAppliedDraft: String?,
): DraftApplication {
val draft =
draftText?.trim()?.ifEmpty { null } ?: return DraftApplication(
input = currentInput,
lastAppliedDraft = null,
consumed = false,
)
if (draft == lastAppliedDraft) {
return DraftApplication(
input = currentInput,
lastAppliedDraft = lastAppliedDraft,
consumed = false,
)
}
return DraftApplication(
input = draft,
lastAppliedDraft = draft,
consumed = true,
)
}
@Composable
fun ChatComposer(
draftText: String?,
healthOk: Boolean,
thinkingLevel: String,
pendingRunCount: Int,
attachments: List<PendingImageAttachment>,
onDraftApplied: () -> Unit,
onPickImages: () -> Unit,
onRemoveAttachment: (id: String) -> Unit,
onSetThinkingLevel: (level: String) -> Unit,
onRefresh: () -> Unit,
onAbort: () -> Unit,
onSend: (text: String) -> Unit,
) {
var input by rememberSaveable { mutableStateOf("") }
var lastAppliedDraft by rememberSaveable { mutableStateOf<String?>(null) }
var showThinkingMenu by remember { mutableStateOf(false) }
LaunchedEffect(draftText) {
val next = applyDraftText(draftText = draftText, currentInput = input, lastAppliedDraft = lastAppliedDraft)
input = next.input
lastAppliedDraft = next.lastAppliedDraft
if (next.consumed) {
onDraftApplied()
}
}
val canSend = pendingRunCount == 0 && (input.trim().isNotEmpty() || attachments.isNotEmpty()) && healthOk
val sendBusy = pendingRunCount > 0
Column(modifier = Modifier.fillMaxWidth(), verticalArrangement = Arrangement.spacedBy(8.dp)) {
if (attachments.isNotEmpty()) {
AttachmentsStrip(attachments = attachments, onRemoveAttachment = onRemoveAttachment)
}
OutlinedTextField(
value = input,
onValueChange = { input = it },
modifier = Modifier.fillMaxWidth(),
placeholder = { Text("Type a message…", style = mobileBodyStyle(), color = mobileTextTertiary) },
minLines = 2,
maxLines = 5,
textStyle = mobileBodyStyle().copy(color = mobileText),
shape = RoundedCornerShape(14.dp),
colors = chatTextFieldColors(),
)
if (!healthOk) {
Text(
text = "Gateway is offline. Connect first in the Connect tab.",
style = mobileCallout,
color = ai.openclaw.app.ui.mobileWarning,
)
}
Row(
modifier = Modifier.fillMaxWidth(),
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.spacedBy(8.dp),
) {
Box {
Surface(
onClick = { showThinkingMenu = true },
shape = RoundedCornerShape(14.dp),
color = mobileCardSurface,
border = BorderStroke(1.dp, mobileBorderStrong),
) {
Row(
modifier = Modifier.padding(horizontal = 10.dp, vertical = 10.dp),
verticalAlignment = Alignment.CenterVertically,
) {
Text(
text = thinkingLabel(thinkingLevel),
style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold),
color = mobileTextSecondary,
)
Icon(Icons.Default.ArrowDropDown, contentDescription = "Select thinking level", modifier = Modifier.size(18.dp), tint = mobileTextTertiary)
}
}
DropdownMenu(
expanded = showThinkingMenu,
onDismissRequest = { showThinkingMenu = false },
shape = RoundedCornerShape(16.dp),
containerColor = mobileCardSurface,
tonalElevation = 0.dp,
shadowElevation = 8.dp,
border = BorderStroke(1.dp, mobileBorder),
) {
ThinkingMenuItem("off", thinkingLevel, onSetThinkingLevel) { showThinkingMenu = false }
ThinkingMenuItem("low", thinkingLevel, onSetThinkingLevel) { showThinkingMenu = false }
ThinkingMenuItem("medium", thinkingLevel, onSetThinkingLevel) { showThinkingMenu = false }
ThinkingMenuItem("high", thinkingLevel, onSetThinkingLevel) { showThinkingMenu = false }
}
}
SecondaryActionButton(
label = "Attach",
icon = Icons.Default.AttachFile,
enabled = true,
compact = true,
onClick = onPickImages,
)
SecondaryActionButton(
label = "Refresh",
icon = Icons.Default.Refresh,
enabled = true,
compact = true,
onClick = onRefresh,
)
SecondaryActionButton(
label = "Abort",
icon = Icons.Default.Stop,
enabled = pendingRunCount > 0,
compact = true,
onClick = onAbort,
)
Spacer(modifier = Modifier.weight(1f))
Button(
onClick = {
val text = input
input = ""
onSend(text)
},
enabled = canSend,
modifier = Modifier.height(44.dp),
shape = RoundedCornerShape(14.dp),
contentPadding = PaddingValues(horizontal = 20.dp),
colors =
ButtonDefaults.buttonColors(
containerColor = mobileAccent,
contentColor = Color.White,
disabledContainerColor = mobileBorderStrong,
disabledContentColor = mobileTextTertiary,
),
border = BorderStroke(1.dp, if (canSend) mobileAccentBorderStrong else mobileBorderStrong),
) {
if (sendBusy) {
CircularProgressIndicator(modifier = Modifier.size(16.dp), strokeWidth = 2.dp, color = Color.White)
} else {
Icon(Icons.AutoMirrored.Filled.Send, contentDescription = null, modifier = Modifier.size(16.dp))
}
Spacer(modifier = Modifier.width(6.dp))
Text(
text = "Send",
style = mobileHeadline.copy(fontWeight = FontWeight.Bold),
maxLines = 1,
overflow = TextOverflow.Ellipsis,
)
}
}
}
}
@Composable
private fun SecondaryActionButton(
label: String,
icon: androidx.compose.ui.graphics.vector.ImageVector,
enabled: Boolean,
compact: Boolean = false,
onClick: () -> Unit,
) {
Button(
onClick = onClick,
enabled = enabled,
modifier = if (compact) Modifier.size(44.dp) else Modifier.height(44.dp),
shape = RoundedCornerShape(14.dp),
colors =
ButtonDefaults.buttonColors(
containerColor = mobileCardSurface,
contentColor = mobileTextSecondary,
disabledContainerColor = mobileCardSurface,
disabledContentColor = mobileTextTertiary,
),
border = BorderStroke(1.dp, mobileBorderStrong),
contentPadding = if (compact) PaddingValues(0.dp) else ButtonDefaults.ContentPadding,
) {
Icon(icon, contentDescription = label, modifier = Modifier.size(14.dp))
if (!compact) {
Spacer(modifier = Modifier.width(5.dp))
Text(
text = label,
style = mobileCallout.copy(fontWeight = FontWeight.SemiBold),
color = if (enabled) mobileTextSecondary else mobileTextTertiary,
)
}
}
}
@Composable
private fun ThinkingMenuItem(
value: String,
current: String,
onSet: (String) -> Unit,
onDismiss: () -> Unit,
) {
DropdownMenuItem(
text = { Text(thinkingLabel(value), style = mobileCallout, color = mobileText) },
onClick = {
onSet(value)
onDismiss()
},
trailingIcon = {
if (value == current.trim().lowercase()) {
Text("✓", style = mobileCallout, color = mobileAccent)
} else {
Spacer(modifier = Modifier.width(10.dp))
}
},
)
}
private fun thinkingLabel(raw: String): String {
return when (raw.trim().lowercase()) {
"low" -> "Low"
"medium" -> "Medium"
"high" -> "High"
else -> "Off"
}
}
@Composable
private fun AttachmentsStrip(
attachments: List<PendingImageAttachment>,
onRemoveAttachment: (id: String) -> Unit,
) {
Row(
modifier = Modifier.fillMaxWidth().horizontalScroll(rememberScrollState()),
horizontalArrangement = Arrangement.spacedBy(8.dp),
) {
for (att in attachments) {
AttachmentChip(
fileName = att.fileName,
onRemove = { onRemoveAttachment(att.id) },
)
}
}
}
@Composable
private fun AttachmentChip(fileName: String, onRemove: () -> Unit) {
Surface(
shape = RoundedCornerShape(999.dp),
color = mobileAccentSoft,
border = BorderStroke(1.dp, mobileBorderStrong),
) {
Row(
modifier = Modifier.padding(horizontal = 10.dp, vertical = 6.dp),
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.spacedBy(8.dp),
) {
Text(
text = fileName,
style = mobileCaption1,
color = mobileText,
maxLines = 1,
overflow = TextOverflow.Ellipsis,
)
Surface(
onClick = onRemove,
shape = RoundedCornerShape(999.dp),
color = mobileCardSurface,
border = BorderStroke(1.dp, mobileBorderStrong),
) {
Text(
text = "×",
style = mobileCaption1.copy(fontWeight = FontWeight.Bold),
color = mobileTextSecondary,
modifier = Modifier.padding(horizontal = 8.dp, vertical = 2.dp),
)
}
}
}
}
@Composable
private fun chatTextFieldColors() =
OutlinedTextFieldDefaults.colors(
focusedContainerColor = mobileSurface,
unfocusedContainerColor = mobileSurface,
focusedBorderColor = mobileAccent,
unfocusedBorderColor = mobileBorder,
focusedTextColor = mobileText,
unfocusedTextColor = mobileText,
cursorColor = mobileAccent,
)
@Composable
private fun mobileBodyStyle() =
MaterialTheme.typography.bodyMedium.copy(
fontFamily = ai.openclaw.app.ui.mobileFontFamily,
fontWeight = FontWeight.Medium,
fontSize = 15.sp,
lineHeight = 22.sp,
)

View file

@ -0,0 +1,150 @@
package ai.openclaw.app.ui.chat
import android.content.ContentResolver
import android.graphics.Bitmap
import android.graphics.BitmapFactory
import android.net.Uri
import android.util.Base64
import android.util.LruCache
import androidx.core.graphics.scale
import ai.openclaw.app.node.JpegSizeLimiter
import java.io.ByteArrayOutputStream
import kotlin.math.max
import kotlin.math.roundToInt
private const val CHAT_ATTACHMENT_MAX_WIDTH = 1600
private const val CHAT_ATTACHMENT_MAX_BASE64_CHARS = 300 * 1024
private const val CHAT_ATTACHMENT_START_QUALITY = 85
private const val CHAT_DECODE_MAX_DIMENSION = 1600
private const val CHAT_IMAGE_CACHE_BYTES = 16 * 1024 * 1024
private val decodedBitmapCache =
object : LruCache<String, Bitmap>(CHAT_IMAGE_CACHE_BYTES) {
override fun sizeOf(key: String, value: Bitmap): Int = value.byteCount.coerceAtLeast(1)
}
internal fun loadSizedImageAttachment(resolver: ContentResolver, uri: Uri): PendingImageAttachment {
val fileName = normalizeAttachmentFileName((uri.lastPathSegment ?: "image").substringAfterLast('/'))
val bitmap = decodeScaledBitmap(resolver, uri, maxDimension = CHAT_ATTACHMENT_MAX_WIDTH)
if (bitmap == null) {
throw IllegalStateException("unsupported attachment")
}
val maxBytes = (CHAT_ATTACHMENT_MAX_BASE64_CHARS / 4) * 3
val encoded =
JpegSizeLimiter.compressToLimit(
initialWidth = bitmap.width,
initialHeight = bitmap.height,
startQuality = CHAT_ATTACHMENT_START_QUALITY,
maxBytes = maxBytes,
minSize = 240,
encode = { width, height, quality ->
val working =
if (width == bitmap.width && height == bitmap.height) {
bitmap
} else {
bitmap.scale(width, height, true)
}
try {
val out = ByteArrayOutputStream()
if (!working.compress(Bitmap.CompressFormat.JPEG, quality, out)) {
throw IllegalStateException("attachment encode failed")
}
out.toByteArray()
} finally {
if (working !== bitmap) {
working.recycle()
}
}
},
)
val base64 = Base64.encodeToString(encoded.bytes, Base64.NO_WRAP)
return PendingImageAttachment(
id = uri.toString() + "#" + System.currentTimeMillis().toString(),
fileName = fileName,
mimeType = "image/jpeg",
base64 = base64,
)
}
internal fun decodeBase64Bitmap(base64: String, maxDimension: Int = CHAT_DECODE_MAX_DIMENSION): Bitmap? {
val cacheKey = "$maxDimension:${base64.length}:${base64.hashCode()}"
decodedBitmapCache.get(cacheKey)?.let { return it }
val bytes = Base64.decode(base64, Base64.DEFAULT)
if (bytes.isEmpty()) return null
val bounds = BitmapFactory.Options().apply { inJustDecodeBounds = true }
BitmapFactory.decodeByteArray(bytes, 0, bytes.size, bounds)
if (bounds.outWidth <= 0 || bounds.outHeight <= 0) return null
val bitmap =
BitmapFactory.decodeByteArray(
bytes,
0,
bytes.size,
BitmapFactory.Options().apply {
inSampleSize = computeInSampleSize(bounds.outWidth, bounds.outHeight, maxDimension)
inPreferredConfig = Bitmap.Config.RGB_565
},
) ?: return null
decodedBitmapCache.put(cacheKey, bitmap)
return bitmap
}
internal fun computeInSampleSize(width: Int, height: Int, maxDimension: Int): Int {
if (width <= 0 || height <= 0 || maxDimension <= 0) return 1
var sample = 1
var longestEdge = max(width, height)
while (longestEdge > maxDimension && sample < 64) {
sample *= 2
longestEdge = max(width / sample, height / sample)
}
return sample.coerceAtLeast(1)
}
internal fun normalizeAttachmentFileName(raw: String): String {
val trimmed = raw.trim()
if (trimmed.isEmpty()) return "image.jpg"
val stem = trimmed.substringBeforeLast('.', missingDelimiterValue = trimmed).ifEmpty { "image" }
return "$stem.jpg"
}
private fun decodeScaledBitmap(
resolver: ContentResolver,
uri: Uri,
maxDimension: Int,
): Bitmap? {
val bounds = BitmapFactory.Options().apply { inJustDecodeBounds = true }
resolver.openInputStream(uri).use { input ->
if (input == null) return null
BitmapFactory.decodeStream(input, null, bounds)
}
if (bounds.outWidth <= 0 || bounds.outHeight <= 0) return null
val decoded =
resolver.openInputStream(uri).use { input ->
if (input == null) return null
BitmapFactory.decodeStream(
input,
null,
BitmapFactory.Options().apply {
inSampleSize = computeInSampleSize(bounds.outWidth, bounds.outHeight, maxDimension)
inPreferredConfig = Bitmap.Config.ARGB_8888
},
)
} ?: return null
val longestEdge = max(decoded.width, decoded.height)
if (longestEdge <= maxDimension) return decoded
val scale = maxDimension.toDouble() / longestEdge.toDouble()
val targetWidth = max(1, (decoded.width * scale).roundToInt())
val targetHeight = max(1, (decoded.height * scale).roundToInt())
val scaled = decoded.scale(targetWidth, targetHeight, true)
if (scaled !== decoded) {
decoded.recycle()
}
return scaled
}

View file

@ -0,0 +1,571 @@
package ai.openclaw.app.ui.chat
import androidx.compose.foundation.Image
import androidx.compose.foundation.background
import androidx.compose.foundation.border
import androidx.compose.foundation.horizontalScroll
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.IntrinsicSize
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.fillMaxHeight
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.height
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.width
import androidx.compose.foundation.rememberScrollState
import androidx.compose.foundation.text.selection.SelectionContainer
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.remember
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.layout.ContentScale
import androidx.compose.ui.text.AnnotatedString
import androidx.compose.ui.text.SpanStyle
import androidx.compose.ui.text.TextStyle
import androidx.compose.ui.text.buildAnnotatedString
import androidx.compose.ui.text.font.FontFamily
import androidx.compose.ui.text.font.FontStyle
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.text.style.TextDecoration
import androidx.compose.ui.text.withStyle
import androidx.compose.ui.unit.dp
import androidx.compose.ui.unit.sp
import ai.openclaw.app.ui.mobileAccent
import ai.openclaw.app.ui.mobileCallout
import ai.openclaw.app.ui.mobileCaption1
import ai.openclaw.app.ui.mobileCodeBg
import ai.openclaw.app.ui.mobileCodeText
import ai.openclaw.app.ui.mobileTextSecondary
import org.commonmark.Extension
import org.commonmark.ext.autolink.AutolinkExtension
import org.commonmark.ext.gfm.strikethrough.Strikethrough
import org.commonmark.ext.gfm.strikethrough.StrikethroughExtension
import org.commonmark.ext.gfm.tables.TableBlock
import org.commonmark.ext.gfm.tables.TableBody
import org.commonmark.ext.gfm.tables.TableCell
import org.commonmark.ext.gfm.tables.TableHead
import org.commonmark.ext.gfm.tables.TableRow
import org.commonmark.ext.gfm.tables.TablesExtension
import org.commonmark.ext.task.list.items.TaskListItemMarker
import org.commonmark.ext.task.list.items.TaskListItemsExtension
import org.commonmark.node.BlockQuote
import org.commonmark.node.BulletList
import org.commonmark.node.Code
import org.commonmark.node.Document
import org.commonmark.node.Emphasis
import org.commonmark.node.FencedCodeBlock
import org.commonmark.node.Heading
import org.commonmark.node.HardLineBreak
import org.commonmark.node.HtmlBlock
import org.commonmark.node.HtmlInline
import org.commonmark.node.Image as MarkdownImage
import org.commonmark.node.IndentedCodeBlock
import org.commonmark.node.Link
import org.commonmark.node.ListItem
import org.commonmark.node.Node
import org.commonmark.node.OrderedList
import org.commonmark.node.Paragraph
import org.commonmark.node.SoftLineBreak
import org.commonmark.node.StrongEmphasis
import org.commonmark.node.Text as MarkdownTextNode
import org.commonmark.node.ThematicBreak
import org.commonmark.parser.Parser
private const val LIST_INDENT_DP = 14
private val dataImageRegex = Regex("^data:image/([a-zA-Z0-9+.-]+);base64,([A-Za-z0-9+/=\\n\\r]+)$")
private val markdownParser: Parser by lazy {
val extensions: List<Extension> =
listOf(
AutolinkExtension.create(),
StrikethroughExtension.create(),
TablesExtension.create(),
TaskListItemsExtension.create(),
)
Parser.builder()
.extensions(extensions)
.build()
}
@Composable
fun ChatMarkdown(text: String, textColor: Color) {
val document = remember(text) { markdownParser.parse(text) as Document }
val inlineStyles = InlineStyles(inlineCodeBg = mobileCodeBg, inlineCodeColor = mobileCodeText, linkColor = mobileAccent, baseCallout = mobileCallout)
Column(verticalArrangement = Arrangement.spacedBy(10.dp)) {
RenderMarkdownBlocks(
start = document.firstChild,
textColor = textColor,
inlineStyles = inlineStyles,
listDepth = 0,
)
}
}
@Composable
private fun RenderMarkdownBlocks(
start: Node?,
textColor: Color,
inlineStyles: InlineStyles,
listDepth: Int,
) {
var node = start
while (node != null) {
val current = node
when (current) {
is Paragraph -> {
RenderParagraph(current, textColor = textColor, inlineStyles = inlineStyles)
}
is Heading -> {
val headingText = remember(current) { buildInlineMarkdown(current.firstChild, inlineStyles) }
Text(
text = headingText,
style = headingStyle(current.level, inlineStyles.baseCallout),
color = textColor,
)
}
is FencedCodeBlock -> {
SelectionContainer(modifier = Modifier.fillMaxWidth()) {
ChatCodeBlock(code = current.literal.orEmpty(), language = current.info?.trim()?.ifEmpty { null })
}
}
is IndentedCodeBlock -> {
SelectionContainer(modifier = Modifier.fillMaxWidth()) {
ChatCodeBlock(code = current.literal.orEmpty(), language = null)
}
}
is BlockQuote -> {
Row(
modifier = Modifier
.fillMaxWidth()
.height(IntrinsicSize.Min)
.padding(vertical = 2.dp),
horizontalArrangement = Arrangement.spacedBy(8.dp),
verticalAlignment = Alignment.Top,
) {
Box(
modifier = Modifier
.width(2.dp)
.fillMaxHeight()
.background(mobileTextSecondary.copy(alpha = 0.35f)),
)
Column(
modifier = Modifier.weight(1f),
verticalArrangement = Arrangement.spacedBy(8.dp),
) {
RenderMarkdownBlocks(
start = current.firstChild,
textColor = textColor,
inlineStyles = inlineStyles,
listDepth = listDepth,
)
}
}
}
is BulletList -> {
RenderBulletList(
list = current,
textColor = textColor,
inlineStyles = inlineStyles,
listDepth = listDepth,
)
}
is OrderedList -> {
RenderOrderedList(
list = current,
textColor = textColor,
inlineStyles = inlineStyles,
listDepth = listDepth,
)
}
is TableBlock -> {
RenderTableBlock(
table = current,
textColor = textColor,
inlineStyles = inlineStyles,
)
}
is ThematicBreak -> {
Box(
modifier = Modifier
.fillMaxWidth()
.height(1.dp)
.background(mobileTextSecondary.copy(alpha = 0.25f)),
)
}
is HtmlBlock -> {
val literal = current.literal.orEmpty().trim()
if (literal.isNotEmpty()) {
Text(
text = literal,
style = mobileCallout.copy(fontFamily = FontFamily.Monospace),
color = textColor,
)
}
}
}
node = current.next
}
}
@Composable
private fun RenderParagraph(
paragraph: Paragraph,
textColor: Color,
inlineStyles: InlineStyles,
) {
val standaloneImage = remember(paragraph) { standaloneDataImage(paragraph) }
if (standaloneImage != null) {
InlineBase64Image(base64 = standaloneImage.base64, mimeType = standaloneImage.mimeType)
return
}
val annotated = remember(paragraph) { buildInlineMarkdown(paragraph.firstChild, inlineStyles) }
if (annotated.text.trimEnd().isEmpty()) {
return
}
Text(
text = annotated,
style = inlineStyles.baseCallout,
color = textColor,
)
}
@Composable
private fun RenderBulletList(
list: BulletList,
textColor: Color,
inlineStyles: InlineStyles,
listDepth: Int,
) {
Column(
modifier = Modifier.padding(start = (LIST_INDENT_DP * listDepth).dp),
verticalArrangement = Arrangement.spacedBy(6.dp),
) {
var item = list.firstChild
while (item != null) {
if (item is ListItem) {
RenderListItem(
item = item,
markerText = "•",
textColor = textColor,
inlineStyles = inlineStyles,
listDepth = listDepth,
)
}
item = item.next
}
}
}
@Composable
private fun RenderOrderedList(
list: OrderedList,
textColor: Color,
inlineStyles: InlineStyles,
listDepth: Int,
) {
Column(
modifier = Modifier.padding(start = (LIST_INDENT_DP * listDepth).dp),
verticalArrangement = Arrangement.spacedBy(6.dp),
) {
var index = list.markerStartNumber ?: 1
var item = list.firstChild
while (item != null) {
if (item is ListItem) {
RenderListItem(
item = item,
markerText = "$index.",
textColor = textColor,
inlineStyles = inlineStyles,
listDepth = listDepth,
)
index += 1
}
item = item.next
}
}
}
@Composable
private fun RenderListItem(
item: ListItem,
markerText: String,
textColor: Color,
inlineStyles: InlineStyles,
listDepth: Int,
) {
var contentStart = item.firstChild
var marker = markerText
val task = contentStart as? TaskListItemMarker
if (task != null) {
marker = if (task.isChecked) "☑" else "☐"
contentStart = task.next
}
Row(
modifier = Modifier.fillMaxWidth(),
horizontalArrangement = Arrangement.spacedBy(8.dp),
verticalAlignment = Alignment.Top,
) {
Text(
text = marker,
style = inlineStyles.baseCallout.copy(fontWeight = FontWeight.SemiBold),
color = textColor,
modifier = Modifier.width(24.dp),
)
Column(
modifier = Modifier.weight(1f),
verticalArrangement = Arrangement.spacedBy(4.dp),
) {
RenderMarkdownBlocks(
start = contentStart,
textColor = textColor,
inlineStyles = inlineStyles,
listDepth = listDepth + 1,
)
}
}
}
@Composable
private fun RenderTableBlock(
table: TableBlock,
textColor: Color,
inlineStyles: InlineStyles,
) {
val rows = remember(table) { buildTableRows(table, inlineStyles) }
if (rows.isEmpty()) return
val maxCols = rows.maxOf { row -> row.cells.size }.coerceAtLeast(1)
val scrollState = rememberScrollState()
Column(
modifier = Modifier
.fillMaxWidth()
.horizontalScroll(scrollState)
.border(1.dp, mobileTextSecondary.copy(alpha = 0.25f)),
) {
for (row in rows) {
Row(
modifier = Modifier.fillMaxWidth(),
) {
for (index in 0 until maxCols) {
val cell = row.cells.getOrNull(index) ?: AnnotatedString("")
Text(
text = cell,
style = if (row.isHeader) mobileCaption1.copy(fontWeight = FontWeight.SemiBold) else inlineStyles.baseCallout,
color = textColor,
modifier = Modifier
.border(1.dp, mobileTextSecondary.copy(alpha = 0.22f))
.padding(horizontal = 8.dp, vertical = 6.dp)
.width(160.dp),
)
}
}
}
}
}
private fun buildTableRows(table: TableBlock, inlineStyles: InlineStyles): List<TableRenderRow> {
val rows = mutableListOf<TableRenderRow>()
var child = table.firstChild
while (child != null) {
when (child) {
is TableHead -> rows.addAll(readTableSection(child, isHeader = true, inlineStyles = inlineStyles))
is TableBody -> rows.addAll(readTableSection(child, isHeader = false, inlineStyles = inlineStyles))
is TableRow -> rows.add(readTableRow(child, isHeader = false, inlineStyles = inlineStyles))
}
child = child.next
}
return rows
}
private fun readTableSection(section: Node, isHeader: Boolean, inlineStyles: InlineStyles): List<TableRenderRow> {
val rows = mutableListOf<TableRenderRow>()
var row = section.firstChild
while (row != null) {
if (row is TableRow) {
rows.add(readTableRow(row, isHeader = isHeader, inlineStyles = inlineStyles))
}
row = row.next
}
return rows
}
private fun readTableRow(row: TableRow, isHeader: Boolean, inlineStyles: InlineStyles): TableRenderRow {
val cells = mutableListOf<AnnotatedString>()
var cellNode = row.firstChild
while (cellNode != null) {
if (cellNode is TableCell) {
cells.add(buildInlineMarkdown(cellNode.firstChild, inlineStyles))
}
cellNode = cellNode.next
}
return TableRenderRow(isHeader = isHeader, cells = cells)
}
private fun buildInlineMarkdown(start: Node?, inlineStyles: InlineStyles): AnnotatedString {
return buildAnnotatedString {
appendInlineNode(
node = start,
inlineCodeBg = inlineStyles.inlineCodeBg,
inlineCodeColor = inlineStyles.inlineCodeColor,
linkColor = inlineStyles.linkColor,
)
}
}
private fun AnnotatedString.Builder.appendInlineNode(
node: Node?,
inlineCodeBg: Color,
inlineCodeColor: Color,
linkColor: Color,
) {
var current = node
while (current != null) {
when (current) {
is MarkdownTextNode -> append(current.literal)
is SoftLineBreak -> append('\n')
is HardLineBreak -> append('\n')
is Code -> {
withStyle(
SpanStyle(
fontFamily = FontFamily.Monospace,
background = inlineCodeBg,
color = inlineCodeColor,
),
) {
append(current.literal)
}
}
is Emphasis -> {
withStyle(SpanStyle(fontStyle = FontStyle.Italic)) {
appendInlineNode(current.firstChild, inlineCodeBg = inlineCodeBg, inlineCodeColor = inlineCodeColor, linkColor = linkColor)
}
}
is StrongEmphasis -> {
withStyle(SpanStyle(fontWeight = FontWeight.SemiBold)) {
appendInlineNode(current.firstChild, inlineCodeBg = inlineCodeBg, inlineCodeColor = inlineCodeColor, linkColor = linkColor)
}
}
is Strikethrough -> {
withStyle(SpanStyle(textDecoration = TextDecoration.LineThrough)) {
appendInlineNode(current.firstChild, inlineCodeBg = inlineCodeBg, inlineCodeColor = inlineCodeColor, linkColor = linkColor)
}
}
is Link -> {
withStyle(
SpanStyle(
color = linkColor,
textDecoration = TextDecoration.Underline,
),
) {
appendInlineNode(current.firstChild, inlineCodeBg = inlineCodeBg, inlineCodeColor = inlineCodeColor, linkColor = linkColor)
}
}
is MarkdownImage -> {
val alt = buildPlainText(current.firstChild)
if (alt.isNotBlank()) {
append(alt)
} else {
append("image")
}
}
is HtmlInline -> {
if (!current.literal.isNullOrBlank()) {
append(current.literal)
}
}
else -> {
appendInlineNode(current.firstChild, inlineCodeBg = inlineCodeBg, inlineCodeColor = inlineCodeColor, linkColor = linkColor)
}
}
current = current.next
}
}
private fun buildPlainText(start: Node?): String {
val sb = StringBuilder()
var node = start
while (node != null) {
when (node) {
is MarkdownTextNode -> sb.append(node.literal)
is SoftLineBreak, is HardLineBreak -> sb.append('\n')
else -> sb.append(buildPlainText(node.firstChild))
}
node = node.next
}
return sb.toString()
}
private fun standaloneDataImage(paragraph: Paragraph): ParsedDataImage? {
val only = paragraph.firstChild as? MarkdownImage ?: return null
if (only.next != null) return null
return parseDataImageDestination(only.destination)
}
private fun parseDataImageDestination(destination: String?): ParsedDataImage? {
val raw = destination?.trim().orEmpty()
if (raw.isEmpty()) return null
val match = dataImageRegex.matchEntire(raw) ?: return null
val subtype = match.groupValues.getOrNull(1)?.trim()?.ifEmpty { "png" } ?: "png"
val base64 = match.groupValues.getOrNull(2)?.replace("\n", "")?.replace("\r", "")?.trim().orEmpty()
if (base64.isEmpty()) return null
return ParsedDataImage(mimeType = "image/$subtype", base64 = base64)
}
private fun headingStyle(level: Int, baseCallout: TextStyle): TextStyle {
return when (level.coerceIn(1, 6)) {
1 -> baseCallout.copy(fontSize = 22.sp, lineHeight = 28.sp, fontWeight = FontWeight.Bold)
2 -> baseCallout.copy(fontSize = 20.sp, lineHeight = 26.sp, fontWeight = FontWeight.Bold)
3 -> baseCallout.copy(fontSize = 18.sp, lineHeight = 24.sp, fontWeight = FontWeight.SemiBold)
4 -> baseCallout.copy(fontSize = 16.sp, lineHeight = 22.sp, fontWeight = FontWeight.SemiBold)
else -> baseCallout.copy(fontWeight = FontWeight.SemiBold)
}
}
private data class InlineStyles(
val inlineCodeBg: Color,
val inlineCodeColor: Color,
val linkColor: Color,
val baseCallout: TextStyle,
)
private data class TableRenderRow(
val isHeader: Boolean,
val cells: List<AnnotatedString>,
)
private data class ParsedDataImage(
val mimeType: String,
val base64: String,
)
@Composable
private fun InlineBase64Image(base64: String, mimeType: String?) {
val imageState = rememberBase64ImageState(base64)
val image = imageState.image
if (image != null) {
Image(
bitmap = image!!,
contentDescription = mimeType ?: "image",
contentScale = ContentScale.Fit,
modifier = Modifier.fillMaxWidth(),
)
} else if (imageState.failed) {
Text(
text = "Image unavailable",
modifier = Modifier.padding(vertical = 2.dp),
style = mobileCaption1,
color = mobileTextSecondary,
)
}
}

View file

@ -0,0 +1,117 @@
package ai.openclaw.app.ui.chat
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.lazy.LazyColumn
import androidx.compose.foundation.lazy.items
import androidx.compose.foundation.lazy.rememberLazyListState
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.material3.Surface
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.remember
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.unit.dp
import ai.openclaw.app.chat.ChatMessage
import ai.openclaw.app.chat.ChatPendingToolCall
import ai.openclaw.app.ui.mobileBorder
import ai.openclaw.app.ui.mobileCallout
import ai.openclaw.app.ui.mobileCardSurface
import ai.openclaw.app.ui.mobileHeadline
import ai.openclaw.app.ui.mobileText
import ai.openclaw.app.ui.mobileTextSecondary
@Composable
fun ChatMessageListCard(
messages: List<ChatMessage>,
pendingRunCount: Int,
pendingToolCalls: List<ChatPendingToolCall>,
streamingAssistantText: String?,
healthOk: Boolean,
modifier: Modifier = Modifier,
) {
val listState = rememberLazyListState()
val displayMessages = remember(messages) { messages.asReversed() }
val stream = streamingAssistantText?.trim()
// New list items/tool rows should animate into view, but token streaming should not restart
// that animation on every delta.
LaunchedEffect(messages.size, pendingRunCount, pendingToolCalls.size) {
listState.animateScrollToItem(index = 0)
}
LaunchedEffect(stream) {
if (!stream.isNullOrEmpty()) {
listState.scrollToItem(index = 0)
}
}
Box(modifier = modifier.fillMaxWidth()) {
LazyColumn(
modifier = Modifier.fillMaxSize(),
state = listState,
reverseLayout = true,
verticalArrangement = Arrangement.spacedBy(10.dp),
contentPadding = androidx.compose.foundation.layout.PaddingValues(bottom = 8.dp),
) {
// With reverseLayout = true, index 0 renders at the BOTTOM.
// So we emit newest items first: streaming → tools → typing → messages (newest→oldest).
if (!stream.isNullOrEmpty()) {
item(key = "stream") {
ChatStreamingAssistantBubble(text = stream)
}
}
if (pendingToolCalls.isNotEmpty()) {
item(key = "tools") {
ChatPendingToolsBubble(toolCalls = pendingToolCalls)
}
}
if (pendingRunCount > 0) {
item(key = "typing") {
ChatTypingIndicatorBubble()
}
}
items(items = displayMessages, key = { it.id }) { message ->
ChatMessageBubble(message = message)
}
}
if (messages.isEmpty() && pendingRunCount == 0 && pendingToolCalls.isEmpty() && streamingAssistantText.isNullOrBlank()) {
EmptyChatHint(modifier = Modifier.align(Alignment.Center), healthOk = healthOk)
}
}
}
@Composable
private fun EmptyChatHint(modifier: Modifier = Modifier, healthOk: Boolean) {
Surface(
modifier = modifier.fillMaxWidth(),
shape = RoundedCornerShape(14.dp),
color = mobileCardSurface.copy(alpha = 0.9f),
border = androidx.compose.foundation.BorderStroke(1.dp, mobileBorder),
) {
androidx.compose.foundation.layout.Column(
modifier = Modifier.padding(horizontal = 12.dp, vertical = 12.dp),
verticalArrangement = Arrangement.spacedBy(4.dp),
) {
Text("No messages yet", style = mobileHeadline, color = mobileText)
Text(
text =
if (healthOk) {
"Send the first prompt to start this session."
} else {
"Connect gateway first, then return to chat."
},
style = mobileCallout,
color = mobileTextSecondary,
)
}
}
}

View file

@ -0,0 +1,302 @@
package ai.openclaw.app.ui.chat
import androidx.compose.foundation.BorderStroke
import androidx.compose.foundation.Image
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.shape.CircleShape
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.material3.Surface
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.remember
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.draw.alpha
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.layout.ContentScale
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.text.font.FontFamily
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.unit.dp
import androidx.compose.ui.unit.sp
import ai.openclaw.app.chat.ChatMessage
import ai.openclaw.app.chat.ChatMessageContent
import ai.openclaw.app.chat.ChatPendingToolCall
import ai.openclaw.app.tools.ToolDisplayRegistry
import ai.openclaw.app.ui.mobileAccent
import ai.openclaw.app.ui.mobileAccentSoft
import ai.openclaw.app.ui.mobileBorder
import ai.openclaw.app.ui.mobileBorderStrong
import ai.openclaw.app.ui.mobileCallout
import ai.openclaw.app.ui.mobileCaption1
import ai.openclaw.app.ui.mobileCaption2
import ai.openclaw.app.ui.mobileCardSurface
import ai.openclaw.app.ui.mobileCodeBg
import ai.openclaw.app.ui.mobileCodeBorder
import ai.openclaw.app.ui.mobileCodeText
import ai.openclaw.app.ui.mobileHeadline
import ai.openclaw.app.ui.mobileText
import ai.openclaw.app.ui.mobileTextSecondary
import ai.openclaw.app.ui.mobileWarning
import ai.openclaw.app.ui.mobileWarningSoft
import java.util.Locale
private data class ChatBubbleStyle(
val alignEnd: Boolean,
val containerColor: Color,
val borderColor: Color,
val roleColor: Color,
)
@Composable
fun ChatMessageBubble(message: ChatMessage) {
val role = message.role.trim().lowercase(Locale.US)
val style = bubbleStyle(role)
// Filter to only displayable content parts (text with content, or base64 images).
val displayableContent =
message.content.filter { part ->
when (part.type) {
"text" -> !part.text.isNullOrBlank()
else -> part.base64 != null
}
}
if (displayableContent.isEmpty()) return
ChatBubbleContainer(style = style, roleLabel = roleLabel(role)) {
ChatMessageBody(content = displayableContent, textColor = mobileText)
}
}
@Composable
private fun ChatBubbleContainer(
style: ChatBubbleStyle,
roleLabel: String,
modifier: Modifier = Modifier,
content: @Composable () -> Unit,
) {
Row(
modifier = modifier.fillMaxWidth(),
horizontalArrangement = if (style.alignEnd) Arrangement.End else Arrangement.Start,
) {
Surface(
shape = RoundedCornerShape(12.dp),
border = BorderStroke(1.dp, style.borderColor),
color = style.containerColor,
tonalElevation = 0.dp,
shadowElevation = 0.dp,
modifier = Modifier.fillMaxWidth(0.90f),
) {
Column(
modifier = Modifier.padding(horizontal = 11.dp, vertical = 8.dp),
verticalArrangement = Arrangement.spacedBy(3.dp),
) {
Text(
text = roleLabel,
style = mobileCaption2.copy(fontWeight = FontWeight.SemiBold, letterSpacing = 0.6.sp),
color = style.roleColor,
)
content()
}
}
}
}
@Composable
private fun ChatMessageBody(content: List<ChatMessageContent>, textColor: Color) {
Column(verticalArrangement = Arrangement.spacedBy(8.dp)) {
for (part in content) {
when (part.type) {
"text" -> {
val text = part.text ?: continue
ChatMarkdown(text = text, textColor = textColor)
}
else -> {
val b64 = part.base64 ?: continue
ChatBase64Image(base64 = b64, mimeType = part.mimeType)
}
}
}
}
}
@Composable
fun ChatTypingIndicatorBubble() {
ChatBubbleContainer(
style = bubbleStyle("assistant"),
roleLabel = roleLabel("assistant"),
) {
Row(
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.spacedBy(8.dp),
) {
DotPulse(color = mobileTextSecondary)
Text("Thinking...", style = mobileCallout, color = mobileTextSecondary)
}
}
}
@Composable
fun ChatPendingToolsBubble(toolCalls: List<ChatPendingToolCall>) {
val context = LocalContext.current
val displays =
remember(toolCalls, context) {
toolCalls.map { ToolDisplayRegistry.resolve(context, it.name, it.args) }
}
ChatBubbleContainer(
style = bubbleStyle("assistant"),
roleLabel = "Tools",
) {
Column(verticalArrangement = Arrangement.spacedBy(4.dp)) {
Text("Running tools...", style = mobileCaption1.copy(fontWeight = FontWeight.SemiBold), color = mobileTextSecondary)
for (display in displays.take(6)) {
Column(verticalArrangement = Arrangement.spacedBy(2.dp)) {
Text(
"${display.emoji} ${display.label}",
style = mobileCallout,
color = mobileTextSecondary,
fontFamily = FontFamily.Monospace,
)
display.detailLine?.let { detail ->
Text(
detail,
style = mobileCaption1,
color = mobileTextSecondary,
fontFamily = FontFamily.Monospace,
)
}
}
}
if (toolCalls.size > 6) {
Text(
text = "... +${toolCalls.size - 6} more",
style = mobileCaption1,
color = mobileTextSecondary,
)
}
}
}
}
@Composable
fun ChatStreamingAssistantBubble(text: String) {
ChatBubbleContainer(
style = bubbleStyle("assistant").copy(borderColor = mobileAccent),
roleLabel = "OpenClaw · Live",
) {
ChatMarkdown(text = text, textColor = mobileText)
}
}
@Composable
private fun bubbleStyle(role: String): ChatBubbleStyle {
return when (role) {
"user" ->
ChatBubbleStyle(
alignEnd = true,
containerColor = mobileAccentSoft,
borderColor = mobileAccent,
roleColor = mobileAccent,
)
"system" ->
ChatBubbleStyle(
alignEnd = false,
containerColor = mobileWarningSoft,
borderColor = mobileWarning.copy(alpha = 0.45f),
roleColor = mobileWarning,
)
else ->
ChatBubbleStyle(
alignEnd = false,
containerColor = mobileCardSurface,
borderColor = mobileBorderStrong,
roleColor = mobileTextSecondary,
)
}
}
private fun roleLabel(role: String): String {
return when (role) {
"user" -> "You"
"system" -> "System"
else -> "OpenClaw"
}
}
@Composable
private fun ChatBase64Image(base64: String, mimeType: String?) {
val imageState = rememberBase64ImageState(base64)
val image = imageState.image
if (image != null) {
Surface(
shape = RoundedCornerShape(10.dp),
border = BorderStroke(1.dp, mobileBorder),
color = mobileCardSurface,
modifier = Modifier.fillMaxWidth(),
) {
Image(
bitmap = image!!,
contentDescription = mimeType ?: "attachment",
contentScale = ContentScale.Fit,
modifier = Modifier.fillMaxWidth(),
)
}
} else if (imageState.failed) {
Text("Unsupported attachment", style = mobileCaption1, color = mobileTextSecondary)
}
}
@Composable
private fun DotPulse(color: Color) {
Row(horizontalArrangement = Arrangement.spacedBy(5.dp), verticalAlignment = Alignment.CenterVertically) {
PulseDot(alpha = 0.38f, color = color)
PulseDot(alpha = 0.62f, color = color)
PulseDot(alpha = 0.90f, color = color)
}
}
@Composable
private fun PulseDot(alpha: Float, color: Color) {
Surface(
modifier = Modifier.size(6.dp).alpha(alpha),
shape = CircleShape,
color = color,
) {}
}
@Composable
fun ChatCodeBlock(code: String, language: String?) {
Surface(
shape = RoundedCornerShape(8.dp),
color = mobileCodeBg,
border = BorderStroke(1.dp, mobileCodeBorder),
modifier = Modifier.fillMaxWidth(),
) {
Column(modifier = Modifier.padding(horizontal = 10.dp, vertical = 8.dp), verticalArrangement = Arrangement.spacedBy(4.dp)) {
if (!language.isNullOrBlank()) {
Text(
text = language.uppercase(Locale.US),
style = mobileCaption2.copy(letterSpacing = 0.4.sp),
color = mobileTextSecondary,
)
}
Text(
text = code.trimEnd(),
fontFamily = FontFamily.Monospace,
style = mobileCallout,
color = mobileCodeText,
)
}
}
}

View file

@ -0,0 +1,260 @@
package ai.openclaw.app.ui.chat
import androidx.activity.compose.rememberLauncherForActivityResult
import androidx.activity.result.contract.ActivityResultContracts
import androidx.compose.foundation.BorderStroke
import androidx.compose.foundation.horizontalScroll
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.imePadding
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.rememberScrollState
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.material3.Surface
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateListOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.rememberCoroutineScope
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.text.style.TextOverflow
import androidx.compose.ui.unit.dp
import androidx.compose.ui.unit.sp
import ai.openclaw.app.MainViewModel
import ai.openclaw.app.chat.ChatSessionEntry
import ai.openclaw.app.chat.OutgoingAttachment
import ai.openclaw.app.ui.mobileAccent
import ai.openclaw.app.ui.mobileAccentBorderStrong
import ai.openclaw.app.ui.mobileBorder
import ai.openclaw.app.ui.mobileBorderStrong
import ai.openclaw.app.ui.mobileCallout
import ai.openclaw.app.ui.mobileCardSurface
import ai.openclaw.app.ui.mobileCaption1
import ai.openclaw.app.ui.mobileCaption2
import ai.openclaw.app.ui.mobileDanger
import ai.openclaw.app.ui.mobileDangerSoft
import ai.openclaw.app.ui.mobileText
import ai.openclaw.app.ui.mobileTextSecondary
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.launch
import kotlinx.coroutines.withContext
internal fun resolvePendingAssistantAutoSend(
pendingPrompt: String?,
healthOk: Boolean,
pendingRunCount: Int,
): String? {
val prompt = pendingPrompt?.trim()?.ifEmpty { null } ?: return null
if (!healthOk || pendingRunCount > 0) return null
return prompt
}
internal suspend fun dispatchPendingAssistantAutoSend(
pendingPrompt: String?,
healthOk: Boolean,
pendingRunCount: Int,
dispatch: suspend (String) -> Boolean,
): Boolean {
val prompt =
resolvePendingAssistantAutoSend(
pendingPrompt = pendingPrompt,
healthOk = healthOk,
pendingRunCount = pendingRunCount,
) ?: return false
return dispatch(prompt)
}
@Composable
fun ChatSheetContent(viewModel: MainViewModel) {
val messages by viewModel.chatMessages.collectAsState()
val errorText by viewModel.chatError.collectAsState()
val pendingRunCount by viewModel.pendingRunCount.collectAsState()
val healthOk by viewModel.chatHealthOk.collectAsState()
val sessionKey by viewModel.chatSessionKey.collectAsState()
val mainSessionKey by viewModel.mainSessionKey.collectAsState()
val thinkingLevel by viewModel.chatThinkingLevel.collectAsState()
val streamingAssistantText by viewModel.chatStreamingAssistantText.collectAsState()
val pendingToolCalls by viewModel.chatPendingToolCalls.collectAsState()
val sessions by viewModel.chatSessions.collectAsState()
val chatDraft by viewModel.chatDraft.collectAsState()
val pendingAssistantAutoSend by viewModel.pendingAssistantAutoSend.collectAsState()
LaunchedEffect(Unit) {
viewModel.loadChat(mainSessionKey)
}
LaunchedEffect(pendingAssistantAutoSend, healthOk, pendingRunCount, thinkingLevel) {
val accepted =
dispatchPendingAssistantAutoSend(
pendingPrompt = pendingAssistantAutoSend,
healthOk = healthOk,
pendingRunCount = pendingRunCount,
) { prompt ->
viewModel.sendChatAwaitAcceptance(
message = prompt,
thinking = thinkingLevel,
attachments = emptyList(),
)
}
if (!accepted) return@LaunchedEffect
viewModel.clearPendingAssistantAutoSend()
}
val context = LocalContext.current
val resolver = context.contentResolver
val scope = rememberCoroutineScope()
val attachments = remember { mutableStateListOf<PendingImageAttachment>() }
val pickImages =
rememberLauncherForActivityResult(ActivityResultContracts.GetMultipleContents()) { uris ->
if (uris.isNullOrEmpty()) return@rememberLauncherForActivityResult
scope.launch(Dispatchers.IO) {
val next =
uris.take(8).mapNotNull { uri ->
try {
loadSizedImageAttachment(resolver, uri)
} catch (_: Throwable) {
null
}
}
withContext(Dispatchers.Main) {
attachments.addAll(next)
}
}
}
Column(
modifier =
Modifier
.fillMaxSize()
.padding(horizontal = 20.dp, vertical = 12.dp),
verticalArrangement = Arrangement.spacedBy(8.dp),
) {
ChatThreadSelector(
sessionKey = sessionKey,
sessions = sessions,
mainSessionKey = mainSessionKey,
onSelectSession = { key -> viewModel.switchChatSession(key) },
)
if (!errorText.isNullOrBlank()) {
ChatErrorRail(errorText = errorText!!)
}
ChatMessageListCard(
messages = messages,
pendingRunCount = pendingRunCount,
pendingToolCalls = pendingToolCalls,
streamingAssistantText = streamingAssistantText,
healthOk = healthOk,
modifier = Modifier.weight(1f, fill = true),
)
Row(modifier = Modifier.fillMaxWidth().imePadding()) {
ChatComposer(
draftText = chatDraft,
healthOk = healthOk,
thinkingLevel = thinkingLevel,
pendingRunCount = pendingRunCount,
attachments = attachments,
onDraftApplied = viewModel::clearChatDraft,
onPickImages = { pickImages.launch("image/*") },
onRemoveAttachment = { id -> attachments.removeAll { it.id == id } },
onSetThinkingLevel = { level -> viewModel.setChatThinkingLevel(level) },
onRefresh = {
viewModel.refreshChat()
viewModel.refreshChatSessions(limit = 200)
},
onAbort = { viewModel.abortChat() },
onSend = { text ->
val outgoing =
attachments.map { att ->
OutgoingAttachment(
type = "image",
mimeType = att.mimeType,
fileName = att.fileName,
base64 = att.base64,
)
}
viewModel.sendChat(message = text, thinking = thinkingLevel, attachments = outgoing)
attachments.clear()
},
)
}
}
}
@Composable
private fun ChatThreadSelector(
sessionKey: String,
sessions: List<ChatSessionEntry>,
mainSessionKey: String,
onSelectSession: (String) -> Unit,
) {
val sessionOptions =
remember(sessionKey, sessions, mainSessionKey) {
resolveSessionChoices(sessionKey, sessions, mainSessionKey = mainSessionKey)
}
Row(
modifier = Modifier.fillMaxWidth().horizontalScroll(rememberScrollState()),
horizontalArrangement = Arrangement.spacedBy(8.dp),
) {
for (entry in sessionOptions) {
val active = entry.key == sessionKey
Surface(
onClick = { onSelectSession(entry.key) },
shape = RoundedCornerShape(14.dp),
color = if (active) mobileAccent else mobileCardSurface,
border = BorderStroke(1.dp, if (active) mobileAccentBorderStrong else mobileBorderStrong),
tonalElevation = 0.dp,
shadowElevation = 0.dp,
) {
Text(
text = friendlySessionName(entry.displayName ?: entry.key),
style = mobileCaption1.copy(fontWeight = if (active) FontWeight.Bold else FontWeight.SemiBold),
color = if (active) Color.White else mobileText,
maxLines = 1,
overflow = TextOverflow.Ellipsis,
modifier = Modifier.padding(horizontal = 12.dp, vertical = 8.dp),
)
}
}
}
}
@Composable
private fun ChatErrorRail(errorText: String) {
Surface(
modifier = Modifier.fillMaxWidth(),
color = mobileDangerSoft,
shape = RoundedCornerShape(12.dp),
border = androidx.compose.foundation.BorderStroke(1.dp, mobileDanger),
) {
Column(modifier = Modifier.padding(horizontal = 10.dp, vertical = 8.dp), verticalArrangement = Arrangement.spacedBy(2.dp)) {
Text(
text = "CHAT ERROR",
style = mobileCaption2.copy(letterSpacing = 0.6.sp),
color = mobileDanger,
)
Text(text = errorText, style = mobileCallout, color = mobileText)
}
}
}
data class PendingImageAttachment(
val id: String,
val fileName: String,
val mimeType: String,
val base64: String,
)

View file

@ -0,0 +1,73 @@
package ai.openclaw.app.ui.chat
import ai.openclaw.app.chat.ChatSessionEntry
private const val RECENT_WINDOW_MS = 24 * 60 * 60 * 1000L
/**
* Derive a human-friendly label from a raw session key.
* Examples:
* "telegram:g-agent-main-main" -> "Main"
* "agent:main:main" -> "Main"
* "discord:g-server-channel" -> "Server Channel"
* "my-custom-session" -> "My Custom Session"
*/
fun friendlySessionName(key: String): String {
// Strip common prefixes like "telegram:", "agent:", "discord:" etc.
val stripped = key.substringAfterLast(":")
// Remove leading "g-" prefix (gateway artifact)
val cleaned = if (stripped.startsWith("g-")) stripped.removePrefix("g-") else stripped
// Split on hyphens/underscores, title-case each word, collapse "main main" -> "Main"
val words = cleaned.split('-', '_').filter { it.isNotBlank() }.map { word ->
word.replaceFirstChar { it.uppercaseChar() }
}.distinct()
val result = words.joinToString(" ")
return result.ifBlank { key }
}
fun resolveSessionChoices(
currentSessionKey: String,
sessions: List<ChatSessionEntry>,
mainSessionKey: String,
nowMs: Long = System.currentTimeMillis(),
): List<ChatSessionEntry> {
val mainKey = mainSessionKey.trim().ifEmpty { "main" }
val current = currentSessionKey.trim().let { if (it == "main" && mainKey != "main") mainKey else it }
val aliasKey = if (mainKey == "main") null else "main"
val cutoff = nowMs - RECENT_WINDOW_MS
val sorted = sessions.sortedByDescending { it.updatedAtMs ?: 0L }
val recent = mutableListOf<ChatSessionEntry>()
val seen = mutableSetOf<String>()
for (entry in sorted) {
if (aliasKey != null && entry.key == aliasKey) continue
if (!seen.add(entry.key)) continue
if ((entry.updatedAtMs ?: 0L) < cutoff) continue
recent.add(entry)
}
val result = mutableListOf<ChatSessionEntry>()
val included = mutableSetOf<String>()
val mainEntry = sorted.firstOrNull { it.key == mainKey }
if (mainEntry != null) {
result.add(mainEntry)
included.add(mainKey)
} else if (current == mainKey) {
result.add(ChatSessionEntry(key = mainKey, updatedAtMs = null))
included.add(mainKey)
}
for (entry in recent) {
if (included.add(entry.key)) {
result.add(entry)
}
}
if (current.isNotEmpty() && !included.contains(current)) {
result.add(ChatSessionEntry(key = current, updatedAtMs = null))
}
return result
}

View file

@ -0,0 +1,695 @@
package ai.openclaw.app.voice
import android.Manifest
import android.content.Context
import android.content.Intent
import android.content.pm.PackageManager
import android.os.Bundle
import android.os.Handler
import android.os.Looper
import android.util.Log
import android.speech.RecognitionListener
import android.speech.RecognizerIntent
import android.speech.SpeechRecognizer
import androidx.core.content.ContextCompat
import java.util.UUID
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.Job
import kotlinx.coroutines.delay
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.launch
import kotlinx.coroutines.withContext
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonArray
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
enum class VoiceConversationRole {
User,
Assistant,
}
data class VoiceConversationEntry(
val id: String,
val role: VoiceConversationRole,
val text: String,
val isStreaming: Boolean = false,
)
class MicCaptureManager(
private val context: Context,
private val scope: CoroutineScope,
/**
* Send [message] to the gateway and return the run ID.
* [onRunIdKnown] is called with the idempotency key *before* the network
* round-trip so [pendingRunId] is set before any chat events can arrive.
*/
private val sendToGateway: suspend (message: String, onRunIdKnown: (String) -> Unit) -> String?,
private val speakAssistantReply: suspend (String) -> Unit = {},
) {
companion object {
private const val tag = "MicCapture"
private const val speechMinSessionMs = 30_000L
private const val speechCompleteSilenceMs = 1_500L
private const val speechPossibleSilenceMs = 900L
private const val transcriptIdleFlushMs = 1_600L
private const val maxConversationEntries = 40
private const val pendingRunTimeoutMs = 45_000L
}
private val mainHandler = Handler(Looper.getMainLooper())
private val json = Json { ignoreUnknownKeys = true }
private val _micEnabled = MutableStateFlow(false)
val micEnabled: StateFlow<Boolean> = _micEnabled
private val _micCooldown = MutableStateFlow(false)
val micCooldown: StateFlow<Boolean> = _micCooldown
private val _isListening = MutableStateFlow(false)
val isListening: StateFlow<Boolean> = _isListening
private val _statusText = MutableStateFlow("Mic off")
val statusText: StateFlow<String> = _statusText
private val _liveTranscript = MutableStateFlow<String?>(null)
val liveTranscript: StateFlow<String?> = _liveTranscript
private val _queuedMessages = MutableStateFlow<List<String>>(emptyList())
val queuedMessages: StateFlow<List<String>> = _queuedMessages
private val _conversation = MutableStateFlow<List<VoiceConversationEntry>>(emptyList())
val conversation: StateFlow<List<VoiceConversationEntry>> = _conversation
private val _inputLevel = MutableStateFlow(0f)
val inputLevel: StateFlow<Float> = _inputLevel
private val _isSending = MutableStateFlow(false)
val isSending: StateFlow<Boolean> = _isSending
private val messageQueue = ArrayDeque<String>()
private val messageQueueLock = Any()
private var flushedPartialTranscript: String? = null
private var pendingRunId: String? = null
private var pendingAssistantEntryId: String? = null
private var gatewayConnected = false
private var recognizer: SpeechRecognizer? = null
private var restartJob: Job? = null
private var drainJob: Job? = null
private var transcriptFlushJob: Job? = null
private var pendingRunTimeoutJob: Job? = null
private var stopRequested = false
private val ttsPauseLock = Any()
private var ttsPauseDepth = 0
private var resumeMicAfterTts = false
private fun enqueueMessage(message: String) {
synchronized(messageQueueLock) {
messageQueue.addLast(message)
}
}
private fun snapshotMessageQueue(): List<String> {
return synchronized(messageQueueLock) {
messageQueue.toList()
}
}
private fun hasQueuedMessages(): Boolean {
return synchronized(messageQueueLock) {
messageQueue.isNotEmpty()
}
}
private fun firstQueuedMessage(): String? {
return synchronized(messageQueueLock) {
messageQueue.firstOrNull()
}
}
private fun removeFirstQueuedMessage(): String? {
return synchronized(messageQueueLock) {
if (messageQueue.isEmpty()) null else messageQueue.removeFirst()
}
}
private fun queuedMessageCount(): Int {
return synchronized(messageQueueLock) {
messageQueue.size
}
}
fun setMicEnabled(enabled: Boolean) {
if (_micEnabled.value == enabled) return
_micEnabled.value = enabled
if (enabled) {
val pausedForTts =
synchronized(ttsPauseLock) {
if (ttsPauseDepth > 0) {
resumeMicAfterTts = true
true
} else {
false
}
}
if (pausedForTts) {
_statusText.value = if (_isSending.value) "Speaking · waiting for reply" else "Speaking…"
return
}
start()
sendQueuedIfIdle()
} else {
// Give the recognizer time to finish processing buffered audio.
// Cancel any prior drain to prevent duplicate sends on rapid toggle.
drainJob?.cancel()
_micCooldown.value = true
drainJob = scope.launch {
delay(2000L)
stop()
// Capture any partial transcript that didn't get a final result from the recognizer
val partial = _liveTranscript.value?.trim().orEmpty()
if (partial.isNotEmpty()) {
queueRecognizedMessage(partial)
}
drainJob = null
_micCooldown.value = false
sendQueuedIfIdle()
}
}
}
suspend fun pauseForTts() {
val shouldPause =
synchronized(ttsPauseLock) {
ttsPauseDepth += 1
if (ttsPauseDepth > 1) return@synchronized false
resumeMicAfterTts = _micEnabled.value
val active = resumeMicAfterTts || recognizer != null || _isListening.value
if (!active) return@synchronized false
stopRequested = true
restartJob?.cancel()
restartJob = null
transcriptFlushJob?.cancel()
transcriptFlushJob = null
_isListening.value = false
_inputLevel.value = 0f
_liveTranscript.value = null
_statusText.value = if (_isSending.value) "Speaking · waiting for reply" else "Speaking…"
true
}
if (!shouldPause) return
withContext(Dispatchers.Main) {
recognizer?.cancel()
recognizer?.destroy()
recognizer = null
}
}
suspend fun resumeAfterTts() {
val shouldResume =
synchronized(ttsPauseLock) {
if (ttsPauseDepth == 0) return@synchronized false
ttsPauseDepth -= 1
if (ttsPauseDepth > 0) return@synchronized false
val resume = resumeMicAfterTts && _micEnabled.value
resumeMicAfterTts = false
if (!resume) {
_statusText.value =
when {
_micEnabled.value && _isSending.value -> "Listening · sending queued voice"
_micEnabled.value -> "Listening"
_isSending.value -> "Mic off · sending…"
else -> "Mic off"
}
}
resume
}
if (!shouldResume) return
stopRequested = false
start()
sendQueuedIfIdle()
}
fun onGatewayConnectionChanged(connected: Boolean) {
gatewayConnected = connected
if (connected) {
sendQueuedIfIdle()
return
}
pendingRunTimeoutJob?.cancel()
pendingRunTimeoutJob = null
pendingRunId = null
pendingAssistantEntryId = null
_isSending.value = false
if (hasQueuedMessages()) {
_statusText.value = queuedWaitingStatus()
}
}
fun handleGatewayEvent(event: String, payloadJson: String?) {
if (event != "chat") return
if (payloadJson.isNullOrBlank()) return
val payload =
try {
json.parseToJsonElement(payloadJson).asObjectOrNull()
} catch (_: Throwable) {
null
} ?: return
val runId = pendingRunId ?: run { Log.d("MicCapture", "no pendingRunId — drop"); return }
val eventRunId = payload["runId"].asStringOrNull() ?: return
if (eventRunId != runId) { Log.d("MicCapture", "runId mismatch: event=$eventRunId pending=$runId"); return }
when (payload["state"].asStringOrNull()) {
"delta" -> {
val deltaText = parseAssistantText(payload)
if (!deltaText.isNullOrBlank()) {
upsertPendingAssistant(text = deltaText.trim(), isStreaming = true)
}
}
"final" -> {
val finalText = parseAssistantText(payload)?.trim().orEmpty()
if (finalText.isNotEmpty()) {
upsertPendingAssistant(text = finalText, isStreaming = false)
playAssistantReplyAsync(finalText)
} else if (pendingAssistantEntryId != null) {
updateConversationEntry(pendingAssistantEntryId!!, text = null, isStreaming = false)
}
completePendingTurn()
}
"error" -> {
val errorMessage = payload["errorMessage"].asStringOrNull()?.trim().orEmpty().ifEmpty { "Voice request failed" }
upsertPendingAssistant(text = errorMessage, isStreaming = false)
completePendingTurn()
}
"aborted" -> {
upsertPendingAssistant(text = "Response aborted", isStreaming = false)
completePendingTurn()
}
}
}
private fun start() {
stopRequested = false
if (!SpeechRecognizer.isRecognitionAvailable(context)) {
_statusText.value = "Speech recognizer unavailable"
_micEnabled.value = false
return
}
if (!hasMicPermission()) {
_statusText.value = "Microphone permission required"
_micEnabled.value = false
return
}
mainHandler.post {
try {
if (recognizer == null) {
recognizer = SpeechRecognizer.createSpeechRecognizer(context).also { it.setRecognitionListener(listener) }
}
startListeningSession()
} catch (err: Throwable) {
_statusText.value = "Start failed: ${err.message ?: err::class.simpleName}"
_micEnabled.value = false
}
}
}
private fun stop() {
stopRequested = true
restartJob?.cancel()
restartJob = null
transcriptFlushJob?.cancel()
transcriptFlushJob = null
_isListening.value = false
_statusText.value = if (_isSending.value) "Mic off · sending…" else "Mic off"
_inputLevel.value = 0f
mainHandler.post {
recognizer?.cancel()
recognizer?.destroy()
recognizer = null
}
}
private fun startListeningSession() {
val recognizerInstance = recognizer ?: return
val intent =
Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH).apply {
putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, RecognizerIntent.LANGUAGE_MODEL_FREE_FORM)
putExtra(RecognizerIntent.EXTRA_PARTIAL_RESULTS, true)
putExtra(RecognizerIntent.EXTRA_MAX_RESULTS, 3)
putExtra(RecognizerIntent.EXTRA_CALLING_PACKAGE, context.packageName)
putExtra(RecognizerIntent.EXTRA_SPEECH_INPUT_MINIMUM_LENGTH_MILLIS, speechMinSessionMs)
putExtra(RecognizerIntent.EXTRA_SPEECH_INPUT_COMPLETE_SILENCE_LENGTH_MILLIS, speechCompleteSilenceMs)
putExtra(
RecognizerIntent.EXTRA_SPEECH_INPUT_POSSIBLY_COMPLETE_SILENCE_LENGTH_MILLIS,
speechPossibleSilenceMs,
)
}
_statusText.value =
when {
_isSending.value -> "Listening · sending queued voice"
hasQueuedMessages() -> "Listening · ${queuedMessageCount()} queued"
else -> "Listening"
}
_isListening.value = true
recognizerInstance.startListening(intent)
}
private fun scheduleRestart(delayMs: Long = 300L) {
if (stopRequested) return
if (!_micEnabled.value) return
restartJob?.cancel()
restartJob =
scope.launch {
delay(delayMs)
mainHandler.post {
if (stopRequested || !_micEnabled.value) return@post
try {
startListeningSession()
} catch (_: Throwable) {
// retry through onError
}
}
}
}
private fun queueRecognizedMessage(text: String) {
val message = text.trim()
_liveTranscript.value = null
if (message.isEmpty()) return
appendConversation(
role = VoiceConversationRole.User,
text = message,
)
enqueueMessage(message)
publishQueue()
}
private fun scheduleTranscriptFlush(expectedText: String) {
transcriptFlushJob?.cancel()
transcriptFlushJob =
scope.launch {
delay(transcriptIdleFlushMs)
if (!_micEnabled.value || _isSending.value) return@launch
val current = _liveTranscript.value?.trim().orEmpty()
if (current.isEmpty() || current != expectedText) return@launch
flushedPartialTranscript = current
queueRecognizedMessage(current)
sendQueuedIfIdle()
}
}
private fun publishQueue() {
_queuedMessages.value = snapshotMessageQueue()
}
private fun sendQueuedIfIdle() {
if (_isSending.value) return
if (!hasQueuedMessages()) {
if (_micEnabled.value) {
_statusText.value = "Listening"
} else {
_statusText.value = "Mic off"
}
return
}
if (!gatewayConnected) {
_statusText.value = queuedWaitingStatus()
return
}
val next = firstQueuedMessage() ?: return
_isSending.value = true
pendingRunTimeoutJob?.cancel()
pendingRunTimeoutJob = null
_statusText.value = if (_micEnabled.value) "Listening · sending queued voice" else "Sending queued voice"
scope.launch {
try {
val runId = sendToGateway(next) { earlyRunId ->
// Called with the idempotency key before chat.send fires so that
// pendingRunId is populated before any chat events can arrive.
pendingRunId = earlyRunId
}
// Update to the real runId if the gateway returned a different one.
if (runId != null && runId != pendingRunId) pendingRunId = runId
if (runId == null) {
pendingRunTimeoutJob?.cancel()
pendingRunTimeoutJob = null
removeFirstQueuedMessage()
publishQueue()
_isSending.value = false
pendingAssistantEntryId = null
sendQueuedIfIdle()
} else {
armPendingRunTimeout(runId)
}
} catch (err: Throwable) {
pendingRunTimeoutJob?.cancel()
pendingRunTimeoutJob = null
_isSending.value = false
pendingRunId = null
pendingAssistantEntryId = null
_statusText.value =
if (!gatewayConnected) {
queuedWaitingStatus()
} else {
"Send failed: ${err.message ?: err::class.simpleName}"
}
}
}
}
private fun armPendingRunTimeout(runId: String) {
pendingRunTimeoutJob?.cancel()
pendingRunTimeoutJob =
scope.launch {
delay(pendingRunTimeoutMs)
if (pendingRunId != runId) return@launch
pendingRunId = null
pendingAssistantEntryId = null
_isSending.value = false
_statusText.value =
if (gatewayConnected) {
"Voice reply timed out; retrying queued turn"
} else {
queuedWaitingStatus()
}
sendQueuedIfIdle()
}
}
private fun completePendingTurn() {
pendingRunTimeoutJob?.cancel()
pendingRunTimeoutJob = null
if (removeFirstQueuedMessage() != null) {
publishQueue()
}
pendingRunId = null
pendingAssistantEntryId = null
_isSending.value = false
sendQueuedIfIdle()
}
private fun queuedWaitingStatus(): String {
return "${queuedMessageCount()} queued · waiting for gateway"
}
private fun appendConversation(
role: VoiceConversationRole,
text: String,
isStreaming: Boolean = false,
): String {
val id = UUID.randomUUID().toString()
_conversation.value =
(_conversation.value + VoiceConversationEntry(id = id, role = role, text = text, isStreaming = isStreaming))
.takeLast(maxConversationEntries)
return id
}
private fun updateConversationEntry(id: String, text: String?, isStreaming: Boolean) {
val current = _conversation.value
if (current.isEmpty()) return
val targetIndex =
when {
current[current.lastIndex].id == id -> current.lastIndex
else -> current.indexOfFirst { it.id == id }
}
if (targetIndex < 0) return
val entry = current[targetIndex]
val updatedText = text ?: entry.text
if (updatedText == entry.text && entry.isStreaming == isStreaming) return
val updated = current.toMutableList()
updated[targetIndex] = entry.copy(text = updatedText, isStreaming = isStreaming)
_conversation.value = updated
}
private fun upsertPendingAssistant(text: String, isStreaming: Boolean) {
val currentId = pendingAssistantEntryId
if (currentId == null) {
pendingAssistantEntryId =
appendConversation(
role = VoiceConversationRole.Assistant,
text = text,
isStreaming = isStreaming,
)
return
}
updateConversationEntry(id = currentId, text = text, isStreaming = isStreaming)
}
private fun playAssistantReplyAsync(text: String) {
val spoken = text.trim()
if (spoken.isEmpty()) return
scope.launch {
try {
speakAssistantReply(spoken)
} catch (err: Throwable) {
Log.w(tag, "assistant speech failed: ${err.message ?: err::class.simpleName}")
}
}
}
private fun disableMic(status: String) {
stopRequested = true
restartJob?.cancel()
restartJob = null
transcriptFlushJob?.cancel()
transcriptFlushJob = null
_micEnabled.value = false
_isListening.value = false
_inputLevel.value = 0f
_statusText.value = status
mainHandler.post {
recognizer?.cancel()
recognizer?.destroy()
recognizer = null
}
}
private fun hasMicPermission(): Boolean {
return (
ContextCompat.checkSelfPermission(context, Manifest.permission.RECORD_AUDIO) ==
PackageManager.PERMISSION_GRANTED
)
}
private fun parseAssistantText(payload: JsonObject): String? {
val message = payload["message"].asObjectOrNull() ?: return null
if (message["role"].asStringOrNull() != "assistant") return null
val content = message["content"] as? JsonArray ?: return null
val parts =
content.mapNotNull { item ->
val obj = item.asObjectOrNull() ?: return@mapNotNull null
if (obj["type"].asStringOrNull() != "text") return@mapNotNull null
obj["text"].asStringOrNull()?.trim()?.takeIf { it.isNotEmpty() }
}
if (parts.isEmpty()) return null
return parts.joinToString("\n")
}
private val listener =
object : RecognitionListener {
override fun onReadyForSpeech(params: Bundle?) {
_isListening.value = true
}
override fun onBeginningOfSpeech() {}
override fun onRmsChanged(rmsdB: Float) {
val level = ((rmsdB + 2f) / 12f).coerceIn(0f, 1f)
_inputLevel.value = level
}
override fun onBufferReceived(buffer: ByteArray?) {}
override fun onEndOfSpeech() {
_inputLevel.value = 0f
scheduleRestart()
}
override fun onError(error: Int) {
if (stopRequested) return
_isListening.value = false
_inputLevel.value = 0f
val status =
when (error) {
SpeechRecognizer.ERROR_AUDIO -> "Audio error"
SpeechRecognizer.ERROR_CLIENT -> "Client error"
SpeechRecognizer.ERROR_NETWORK -> "Network error"
SpeechRecognizer.ERROR_NETWORK_TIMEOUT -> "Network timeout"
SpeechRecognizer.ERROR_NO_MATCH -> "Listening"
SpeechRecognizer.ERROR_RECOGNIZER_BUSY -> "Recognizer busy"
SpeechRecognizer.ERROR_SERVER -> "Server error"
SpeechRecognizer.ERROR_SPEECH_TIMEOUT -> "Listening"
SpeechRecognizer.ERROR_INSUFFICIENT_PERMISSIONS -> "Microphone permission required"
SpeechRecognizer.ERROR_LANGUAGE_NOT_SUPPORTED -> "Language not supported on this device"
SpeechRecognizer.ERROR_LANGUAGE_UNAVAILABLE -> "Language unavailable on this device"
SpeechRecognizer.ERROR_SERVER_DISCONNECTED -> "Speech service disconnected"
SpeechRecognizer.ERROR_TOO_MANY_REQUESTS -> "Speech requests limited; retrying"
else -> "Speech error ($error)"
}
_statusText.value = status
if (
error == SpeechRecognizer.ERROR_INSUFFICIENT_PERMISSIONS ||
error == SpeechRecognizer.ERROR_LANGUAGE_NOT_SUPPORTED ||
error == SpeechRecognizer.ERROR_LANGUAGE_UNAVAILABLE
) {
disableMic(status)
return
}
val restartDelayMs =
when (error) {
SpeechRecognizer.ERROR_NO_MATCH,
SpeechRecognizer.ERROR_SPEECH_TIMEOUT,
-> 1_200L
SpeechRecognizer.ERROR_TOO_MANY_REQUESTS -> 2_500L
else -> 600L
}
scheduleRestart(delayMs = restartDelayMs)
}
override fun onResults(results: Bundle?) {
transcriptFlushJob?.cancel()
transcriptFlushJob = null
val text = results?.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION).orEmpty().firstOrNull()
if (!text.isNullOrBlank()) {
val trimmed = text.trim()
if (trimmed != flushedPartialTranscript) {
queueRecognizedMessage(trimmed)
sendQueuedIfIdle()
} else {
flushedPartialTranscript = null
_liveTranscript.value = null
}
}
scheduleRestart()
}
override fun onPartialResults(partialResults: Bundle?) {
val text = partialResults?.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION).orEmpty().firstOrNull()
if (!text.isNullOrBlank()) {
val trimmed = text.trim()
_liveTranscript.value = trimmed
scheduleTranscriptFlush(trimmed)
}
}
override fun onEvent(eventType: Int, params: Bundle?) {}
}
}
private fun kotlinx.serialization.json.JsonElement?.asObjectOrNull(): JsonObject? =
this as? JsonObject
private fun kotlinx.serialization.json.JsonElement?.asStringOrNull(): String? =
(this as? JsonPrimitive)?.takeIf { it.isString }?.content

View file

@ -0,0 +1,242 @@
package ai.openclaw.app.voice
import android.content.Context
import android.media.AudioAttributes
import android.media.AudioFormat
import android.media.AudioTrack
import android.media.MediaPlayer
import kotlinx.coroutines.CancellationException
import kotlinx.coroutines.CompletableDeferred
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.delay
import kotlinx.coroutines.withContext
import java.io.File
internal class TalkAudioPlayer(
private val context: Context,
) {
private val lock = Any()
private var active: ActivePlayback? = null
suspend fun play(audio: TalkSpeakAudio) {
when (val mode = resolvePlaybackMode(audio)) {
is TalkPlaybackMode.Pcm -> playPcm(audio.bytes, mode.sampleRate)
is TalkPlaybackMode.Compressed -> playCompressed(audio.bytes, mode.fileExtension)
}
}
fun stop() {
synchronized(lock) {
active?.cancel()
active = null
}
}
internal fun resolvePlaybackMode(audio: TalkSpeakAudio): TalkPlaybackMode {
return resolvePlaybackMode(
outputFormat = audio.outputFormat,
mimeType = audio.mimeType,
fileExtension = audio.fileExtension,
)
}
companion object {
internal fun resolvePlaybackMode(
outputFormat: String?,
mimeType: String?,
fileExtension: String?,
): TalkPlaybackMode {
val normalizedOutputFormat = outputFormat?.trim()?.lowercase()
if (normalizedOutputFormat != null) {
val pcmSampleRate = parsePcmSampleRate(normalizedOutputFormat)
if (pcmSampleRate != null) {
return TalkPlaybackMode.Pcm(sampleRate = pcmSampleRate)
}
}
val normalizedMimeType = mimeType?.trim()?.lowercase()
val extension =
normalizeExtension(
fileExtension ?: inferExtension(outputFormat = normalizedOutputFormat, mimeType = normalizedMimeType),
)
if (extension != null) {
return TalkPlaybackMode.Compressed(fileExtension = extension)
}
throw IllegalStateException("Unsupported talk audio format")
}
private fun parsePcmSampleRate(outputFormat: String): Int? {
return when (outputFormat) {
"pcm_16000" -> 16_000
"pcm_22050" -> 22_050
"pcm_24000" -> 24_000
"pcm_44100" -> 44_100
else -> null
}
}
private fun inferExtension(outputFormat: String?, mimeType: String?): String? {
return when {
outputFormat == "mp3" || outputFormat?.startsWith("mp3_") == true || mimeType == "audio/mpeg" -> ".mp3"
outputFormat == "opus" || outputFormat?.startsWith("opus_") == true || mimeType == "audio/ogg" -> ".ogg"
outputFormat?.endsWith("-wav") == true || mimeType == "audio/wav" -> ".wav"
outputFormat?.endsWith("-webm") == true || mimeType == "audio/webm" -> ".webm"
else -> null
}
}
private fun normalizeExtension(value: String?): String? {
val trimmed = value?.trim()?.lowercase().orEmpty()
if (trimmed.isEmpty()) return null
return if (trimmed.startsWith(".")) trimmed else ".$trimmed"
}
}
private suspend fun playPcm(bytes: ByteArray, sampleRate: Int) {
withContext(Dispatchers.IO) {
val minBufferSize =
AudioTrack.getMinBufferSize(
sampleRate,
AudioFormat.CHANNEL_OUT_MONO,
AudioFormat.ENCODING_PCM_16BIT,
)
if (minBufferSize <= 0) {
throw IllegalStateException("AudioTrack buffer unavailable")
}
val track =
AudioTrack.Builder()
.setAudioAttributes(
AudioAttributes.Builder()
.setUsage(AudioAttributes.USAGE_MEDIA)
.setContentType(AudioAttributes.CONTENT_TYPE_SPEECH)
.build(),
)
.setAudioFormat(
AudioFormat.Builder()
.setEncoding(AudioFormat.ENCODING_PCM_16BIT)
.setSampleRate(sampleRate)
.setChannelMask(AudioFormat.CHANNEL_OUT_MONO)
.build(),
)
.setTransferMode(AudioTrack.MODE_STATIC)
.setBufferSizeInBytes(maxOf(minBufferSize, bytes.size))
.build()
val finished = CompletableDeferred<Unit>()
val playback =
ActivePlayback(
cancel = {
finished.completeExceptionally(CancellationException("assistant speech cancelled"))
runCatching { track.pause() }
runCatching { track.flush() }
runCatching { track.stop() }
},
)
register(playback)
try {
val written = track.write(bytes, 0, bytes.size)
if (written != bytes.size) {
throw IllegalStateException("AudioTrack write failed")
}
val totalFrames = bytes.size / 2
track.play()
while (track.playState == AudioTrack.PLAYSTATE_PLAYING) {
if (track.playbackHeadPosition >= totalFrames) {
finished.complete(Unit)
break
}
delay(20)
}
if (!finished.isCompleted) {
finished.complete(Unit)
}
finished.await()
} finally {
clear(playback)
runCatching { track.pause() }
runCatching { track.flush() }
runCatching { track.stop() }
track.release()
}
}
}
private suspend fun playCompressed(bytes: ByteArray, fileExtension: String) {
val tempFile = withContext(Dispatchers.IO) {
File.createTempFile("talk-audio-", fileExtension, context.cacheDir).apply {
writeBytes(bytes)
}
}
try {
val finished = CompletableDeferred<Unit>()
val player =
withContext(Dispatchers.Main) {
MediaPlayer().apply {
setAudioAttributes(
AudioAttributes.Builder()
.setUsage(AudioAttributes.USAGE_MEDIA)
.setContentType(AudioAttributes.CONTENT_TYPE_SPEECH)
.build(),
)
setDataSource(tempFile.absolutePath)
setOnCompletionListener {
finished.complete(Unit)
}
setOnErrorListener { _, what, extra ->
finished.completeExceptionally(IllegalStateException("MediaPlayer error ($what/$extra)"))
true
}
prepare()
}
}
val playback =
ActivePlayback(
cancel = {
finished.completeExceptionally(CancellationException("assistant speech cancelled"))
runCatching { player.stop() }
},
)
register(playback)
try {
withContext(Dispatchers.Main) {
player.start()
}
finished.await()
} finally {
clear(playback)
withContext(Dispatchers.Main) {
runCatching { player.stop() }
player.release()
}
}
} finally {
withContext(Dispatchers.IO) {
tempFile.delete()
}
}
}
private fun register(playback: ActivePlayback) {
synchronized(lock) {
active?.cancel()
active = playback
}
}
private fun clear(playback: ActivePlayback) {
synchronized(lock) {
if (active === playback) {
active = null
}
}
}
}
internal sealed interface TalkPlaybackMode {
data class Pcm(val sampleRate: Int) : TalkPlaybackMode
data class Compressed(val fileExtension: String) : TalkPlaybackMode
}
private class ActivePlayback(
val cancel: () -> Unit,
)

View file

@ -0,0 +1,5 @@
package ai.openclaw.app.voice
internal object TalkDefaults {
const val defaultSilenceTimeoutMs = 700L
}

View file

@ -0,0 +1,191 @@
package ai.openclaw.app.voice
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonElement
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
private val directiveJson = Json { ignoreUnknownKeys = true }
data class TalkDirective(
val voiceId: String? = null,
val modelId: String? = null,
val speed: Double? = null,
val rateWpm: Int? = null,
val stability: Double? = null,
val similarity: Double? = null,
val style: Double? = null,
val speakerBoost: Boolean? = null,
val seed: Long? = null,
val normalize: String? = null,
val language: String? = null,
val outputFormat: String? = null,
val latencyTier: Int? = null,
val once: Boolean? = null,
)
data class TalkDirectiveParseResult(
val directive: TalkDirective?,
val stripped: String,
val unknownKeys: List<String>,
)
object TalkDirectiveParser {
fun parse(text: String): TalkDirectiveParseResult {
val normalized = text.replace("\r\n", "\n")
val lines = normalized.split("\n").toMutableList()
if (lines.isEmpty()) return TalkDirectiveParseResult(null, text, emptyList())
val firstNonEmpty = lines.indexOfFirst { it.trim().isNotEmpty() }
if (firstNonEmpty == -1) return TalkDirectiveParseResult(null, text, emptyList())
val head = lines[firstNonEmpty].trim()
if (!head.startsWith("{") || !head.endsWith("}")) {
return TalkDirectiveParseResult(null, text, emptyList())
}
val obj = parseJsonObject(head) ?: return TalkDirectiveParseResult(null, text, emptyList())
val speakerBoost =
boolValue(obj, listOf("speaker_boost", "speakerBoost"))
?: boolValue(obj, listOf("no_speaker_boost", "noSpeakerBoost"))?.not()
val directive = TalkDirective(
voiceId = stringValue(obj, listOf("voice", "voice_id", "voiceId")),
modelId = stringValue(obj, listOf("model", "model_id", "modelId")),
speed = doubleValue(obj, listOf("speed")),
rateWpm = intValue(obj, listOf("rate", "wpm")),
stability = doubleValue(obj, listOf("stability")),
similarity = doubleValue(obj, listOf("similarity", "similarity_boost", "similarityBoost")),
style = doubleValue(obj, listOf("style")),
speakerBoost = speakerBoost,
seed = longValue(obj, listOf("seed")),
normalize = stringValue(obj, listOf("normalize", "apply_text_normalization")),
language = stringValue(obj, listOf("lang", "language_code", "language")),
outputFormat = stringValue(obj, listOf("output_format", "format")),
latencyTier = intValue(obj, listOf("latency", "latency_tier", "latencyTier")),
once = boolValue(obj, listOf("once")),
)
val hasDirective = listOf(
directive.voiceId,
directive.modelId,
directive.speed,
directive.rateWpm,
directive.stability,
directive.similarity,
directive.style,
directive.speakerBoost,
directive.seed,
directive.normalize,
directive.language,
directive.outputFormat,
directive.latencyTier,
directive.once,
).any { it != null }
if (!hasDirective) return TalkDirectiveParseResult(null, text, emptyList())
val knownKeys = setOf(
"voice", "voice_id", "voiceid",
"model", "model_id", "modelid",
"speed", "rate", "wpm",
"stability", "similarity", "similarity_boost", "similarityboost",
"style",
"speaker_boost", "speakerboost",
"no_speaker_boost", "nospeakerboost",
"seed",
"normalize", "apply_text_normalization",
"lang", "language_code", "language",
"output_format", "format",
"latency", "latency_tier", "latencytier",
"once",
)
val unknownKeys = obj.keys.filter { !knownKeys.contains(it.lowercase()) }.sorted()
lines.removeAt(firstNonEmpty)
if (firstNonEmpty < lines.size) {
if (lines[firstNonEmpty].trim().isEmpty()) {
lines.removeAt(firstNonEmpty)
}
}
return TalkDirectiveParseResult(directive, lines.joinToString("\n"), unknownKeys)
}
private fun parseJsonObject(line: String): JsonObject? {
return try {
directiveJson.parseToJsonElement(line) as? JsonObject
} catch (_: Throwable) {
null
}
}
private fun stringValue(obj: JsonObject, keys: List<String>): String? {
for (key in keys) {
val value = obj[key].asStringOrNull()?.trim()
if (!value.isNullOrEmpty()) return value
}
return null
}
private fun doubleValue(obj: JsonObject, keys: List<String>): Double? {
for (key in keys) {
val value = obj[key].asDoubleOrNull()
if (value != null) return value
}
return null
}
private fun intValue(obj: JsonObject, keys: List<String>): Int? {
for (key in keys) {
val value = obj[key].asIntOrNull()
if (value != null) return value
}
return null
}
private fun longValue(obj: JsonObject, keys: List<String>): Long? {
for (key in keys) {
val value = obj[key].asLongOrNull()
if (value != null) return value
}
return null
}
private fun boolValue(obj: JsonObject, keys: List<String>): Boolean? {
for (key in keys) {
val value = obj[key].asBooleanOrNull()
if (value != null) return value
}
return null
}
}
private fun JsonElement?.asStringOrNull(): String? =
(this as? JsonPrimitive)?.takeIf { it.isString }?.content
private fun JsonElement?.asDoubleOrNull(): Double? {
val primitive = this as? JsonPrimitive ?: return null
return primitive.content.toDoubleOrNull()
}
private fun JsonElement?.asIntOrNull(): Int? {
val primitive = this as? JsonPrimitive ?: return null
return primitive.content.toIntOrNull()
}
private fun JsonElement?.asLongOrNull(): Long? {
val primitive = this as? JsonPrimitive ?: return null
return primitive.content.toLongOrNull()
}
private fun JsonElement?.asBooleanOrNull(): Boolean? {
val primitive = this as? JsonPrimitive ?: return null
val content = primitive.content.trim().lowercase()
return when (content) {
"true", "yes", "1" -> true
"false", "no", "0" -> false
else -> null
}
}

View file

@ -0,0 +1,50 @@
package ai.openclaw.app.voice
import ai.openclaw.app.normalizeMainKey
import kotlinx.serialization.json.JsonElement
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.booleanOrNull
import kotlinx.serialization.json.contentOrNull
internal data class TalkModeGatewayConfigState(
val mainSessionKey: String,
val interruptOnSpeech: Boolean?,
val silenceTimeoutMs: Long,
)
internal object TalkModeGatewayConfigParser {
fun parse(config: JsonObject?): TalkModeGatewayConfigState {
val talk = config?.get("talk").asObjectOrNull()
val sessionCfg = config?.get("session").asObjectOrNull()
return TalkModeGatewayConfigState(
mainSessionKey = normalizeMainKey(sessionCfg?.get("mainKey").asStringOrNull()),
interruptOnSpeech = talk?.get("interruptOnSpeech").asBooleanOrNull(),
silenceTimeoutMs = resolvedSilenceTimeoutMs(talk),
)
}
fun resolvedSilenceTimeoutMs(talk: JsonObject?): Long {
val fallback = TalkDefaults.defaultSilenceTimeoutMs
val primitive = talk?.get("silenceTimeoutMs") as? JsonPrimitive ?: return fallback
if (primitive.isString) return fallback
val timeout = primitive.content.toDoubleOrNull() ?: return fallback
if (timeout <= 0 || timeout % 1.0 != 0.0 || timeout > Long.MAX_VALUE.toDouble()) {
return fallback
}
return timeout.toLong()
}
}
private fun JsonElement?.asStringOrNull(): String? =
this?.let { element ->
element as? JsonPrimitive
}?.contentOrNull
private fun JsonElement?.asBooleanOrNull(): Boolean? {
val primitive = this as? JsonPrimitive ?: return null
return primitive.booleanOrNull
}
private fun JsonElement?.asObjectOrNull(): JsonObject? =
this as? JsonObject

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,143 @@
package ai.openclaw.app.voice
import ai.openclaw.app.gateway.GatewaySession
import kotlinx.serialization.Serializable
import kotlinx.serialization.encodeToString
import kotlinx.serialization.json.Json
internal data class TalkSpeakAudio(
val bytes: ByteArray,
val provider: String,
val outputFormat: String?,
val voiceCompatible: Boolean?,
val mimeType: String?,
val fileExtension: String?,
)
internal sealed interface TalkSpeakResult {
data class Success(val audio: TalkSpeakAudio) : TalkSpeakResult
data class FallbackToLocal(val message: String) : TalkSpeakResult
data class Failure(val message: String) : TalkSpeakResult
}
internal class TalkSpeakClient(
private val session: GatewaySession? = null,
private val json: Json = Json { ignoreUnknownKeys = true },
private val requestDetailed: (suspend (String, String, Long) -> GatewaySession.RpcResult)? = null,
) {
suspend fun synthesize(text: String, directive: TalkDirective?): TalkSpeakResult {
val response =
try {
performRequest(
method = "talk.speak",
paramsJson = json.encodeToString(TalkSpeakRequest.from(text = text, directive = directive)),
timeoutMs = 45_000,
)
} catch (err: Throwable) {
return TalkSpeakResult.Failure(err.message ?: "talk.speak request failed")
}
if (!response.ok) {
val error = response.error
val message = error?.message ?: "talk.speak request failed"
return if (isFallbackEligible(error)) {
TalkSpeakResult.FallbackToLocal(message)
} else {
TalkSpeakResult.Failure(message)
}
}
val payload =
try {
json.decodeFromString<TalkSpeakResponse>(response.payloadJson ?: "")
} catch (err: Throwable) {
return TalkSpeakResult.Failure(err.message ?: "talk.speak payload invalid")
}
val bytes =
try {
android.util.Base64.decode(payload.audioBase64, android.util.Base64.DEFAULT)
} catch (err: Throwable) {
return TalkSpeakResult.Failure(err.message ?: "talk.speak audio decode failed")
}
if (bytes.isEmpty()) {
return TalkSpeakResult.Failure("talk.speak returned empty audio")
}
return TalkSpeakResult.Success(
TalkSpeakAudio(
bytes = bytes,
provider = payload.provider,
outputFormat = payload.outputFormat,
voiceCompatible = payload.voiceCompatible,
mimeType = payload.mimeType,
fileExtension = payload.fileExtension,
),
)
}
private fun isFallbackEligible(error: GatewaySession.ErrorShape?): Boolean {
val reason = error?.details?.reason
if (reason == null) return true
return reason == "talk_unconfigured" ||
reason == "talk_provider_unsupported" ||
reason == "method_unavailable"
}
private suspend fun performRequest(
method: String,
paramsJson: String,
timeoutMs: Long,
): GatewaySession.RpcResult {
requestDetailed?.let { return it(method, paramsJson, timeoutMs) }
val activeSession = session ?: throw IllegalStateException("session missing")
return activeSession.requestDetailed(method = method, paramsJson = paramsJson, timeoutMs = timeoutMs)
}
}
@Serializable
internal data class TalkSpeakRequest(
val text: String,
val voiceId: String? = null,
val modelId: String? = null,
val outputFormat: String? = null,
val speed: Double? = null,
val rateWpm: Int? = null,
val stability: Double? = null,
val similarity: Double? = null,
val style: Double? = null,
val speakerBoost: Boolean? = null,
val seed: Long? = null,
val normalize: String? = null,
val language: String? = null,
val latencyTier: Int? = null,
) {
companion object {
fun from(text: String, directive: TalkDirective?): TalkSpeakRequest {
return TalkSpeakRequest(
text = text,
voiceId = directive?.voiceId,
modelId = directive?.modelId,
outputFormat = directive?.outputFormat,
speed = directive?.speed,
rateWpm = directive?.rateWpm,
stability = directive?.stability,
similarity = directive?.similarity,
style = directive?.style,
speakerBoost = directive?.speakerBoost,
seed = directive?.seed,
normalize = directive?.normalize,
language = directive?.language,
latencyTier = directive?.latencyTier,
)
}
}
}
@Serializable
private data class TalkSpeakResponse(
val audioBase64: String,
val provider: String,
val outputFormat: String? = null,
val voiceCompatible: Boolean? = null,
val mimeType: String? = null,
val fileExtension: String? = null,
)

View file

@ -0,0 +1,40 @@
package ai.openclaw.app.voice
object VoiceWakeCommandExtractor {
fun extractCommand(text: String, triggerWords: List<String>): String? {
val raw = text.trim()
if (raw.isEmpty()) return null
val triggers =
triggerWords
.map { it.trim().lowercase() }
.filter { it.isNotEmpty() }
.distinct()
if (triggers.isEmpty()) return null
val alternation = triggers.joinToString("|") { Regex.escape(it) }
// Match: "<anything> <trigger><punct/space> <command>"
val regex = Regex("(?i)(?:^|\\s)($alternation)\\b[\\s\\p{Punct}]*([\\s\\S]+)$")
val match = regex.find(raw) ?: return null
val extracted = match.groupValues.getOrNull(2)?.trim().orEmpty()
if (extracted.isEmpty()) return null
val cleaned = extracted.trimStart { it.isWhitespace() || it.isPunctuation() }.trim()
if (cleaned.isEmpty()) return null
return cleaned
}
}
private fun Char.isPunctuation(): Boolean {
return when (Character.getType(this)) {
Character.CONNECTOR_PUNCTUATION.toInt(),
Character.DASH_PUNCTUATION.toInt(),
Character.START_PUNCTUATION.toInt(),
Character.END_PUNCTUATION.toInt(),
Character.INITIAL_QUOTE_PUNCTUATION.toInt(),
Character.FINAL_QUOTE_PUNCTUATION.toInt(),
Character.OTHER_PUNCTUATION.toInt(),
-> true
else -> false
}
}

View file

@ -0,0 +1,173 @@
package ai.openclaw.app.voice
import android.content.Context
import android.content.Intent
import android.os.Bundle
import android.os.Handler
import android.os.Looper
import android.speech.RecognitionListener
import android.speech.RecognizerIntent
import android.speech.SpeechRecognizer
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Job
import kotlinx.coroutines.delay
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.launch
class VoiceWakeManager(
private val context: Context,
private val scope: CoroutineScope,
private val onCommand: suspend (String) -> Unit,
) {
private val mainHandler = Handler(Looper.getMainLooper())
private val _isListening = MutableStateFlow(false)
val isListening: StateFlow<Boolean> = _isListening
private val _statusText = MutableStateFlow("Off")
val statusText: StateFlow<String> = _statusText
var triggerWords: List<String> = emptyList()
private set
private var recognizer: SpeechRecognizer? = null
private var restartJob: Job? = null
private var lastDispatched: String? = null
private var stopRequested = false
fun setTriggerWords(words: List<String>) {
triggerWords = words
}
fun start() {
mainHandler.post {
if (_isListening.value) return@post
stopRequested = false
if (!SpeechRecognizer.isRecognitionAvailable(context)) {
_isListening.value = false
_statusText.value = "Speech recognizer unavailable"
return@post
}
try {
recognizer?.destroy()
recognizer = SpeechRecognizer.createSpeechRecognizer(context).also { it.setRecognitionListener(listener) }
startListeningInternal()
} catch (err: Throwable) {
_isListening.value = false
_statusText.value = "Start failed: ${err.message ?: err::class.simpleName}"
}
}
}
fun stop(statusText: String = "Off") {
stopRequested = true
restartJob?.cancel()
restartJob = null
mainHandler.post {
_isListening.value = false
_statusText.value = statusText
recognizer?.cancel()
recognizer?.destroy()
recognizer = null
}
}
private fun startListeningInternal() {
val r = recognizer ?: return
val intent =
Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH).apply {
putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, RecognizerIntent.LANGUAGE_MODEL_FREE_FORM)
putExtra(RecognizerIntent.EXTRA_PARTIAL_RESULTS, true)
putExtra(RecognizerIntent.EXTRA_MAX_RESULTS, 3)
putExtra(RecognizerIntent.EXTRA_CALLING_PACKAGE, context.packageName)
}
_statusText.value = "Listening"
_isListening.value = true
r.startListening(intent)
}
private fun scheduleRestart(delayMs: Long = 350) {
if (stopRequested) return
restartJob?.cancel()
restartJob =
scope.launch {
delay(delayMs)
mainHandler.post {
if (stopRequested) return@post
try {
recognizer?.cancel()
startListeningInternal()
} catch (_: Throwable) {
// Will be picked up by onError and retry again.
}
}
}
}
private fun handleTranscription(text: String) {
val command = VoiceWakeCommandExtractor.extractCommand(text, triggerWords) ?: return
if (command == lastDispatched) return
lastDispatched = command
scope.launch { onCommand(command) }
_statusText.value = "Triggered"
scheduleRestart(delayMs = 650)
}
private val listener =
object : RecognitionListener {
override fun onReadyForSpeech(params: Bundle?) {
_statusText.value = "Listening"
}
override fun onBeginningOfSpeech() {}
override fun onRmsChanged(rmsdB: Float) {}
override fun onBufferReceived(buffer: ByteArray?) {}
override fun onEndOfSpeech() {
scheduleRestart()
}
override fun onError(error: Int) {
if (stopRequested) return
_isListening.value = false
if (error == SpeechRecognizer.ERROR_INSUFFICIENT_PERMISSIONS) {
_statusText.value = "Microphone permission required"
return
}
_statusText.value =
when (error) {
SpeechRecognizer.ERROR_AUDIO -> "Audio error"
SpeechRecognizer.ERROR_CLIENT -> "Client error"
SpeechRecognizer.ERROR_NETWORK -> "Network error"
SpeechRecognizer.ERROR_NETWORK_TIMEOUT -> "Network timeout"
SpeechRecognizer.ERROR_NO_MATCH -> "Listening"
SpeechRecognizer.ERROR_RECOGNIZER_BUSY -> "Recognizer busy"
SpeechRecognizer.ERROR_SERVER -> "Server error"
SpeechRecognizer.ERROR_SPEECH_TIMEOUT -> "Listening"
else -> "Speech error ($error)"
}
scheduleRestart(delayMs = 600)
}
override fun onResults(results: Bundle?) {
val list = results?.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION).orEmpty()
list.firstOrNull()?.let(::handleTranscription)
scheduleRestart()
}
override fun onPartialResults(partialResults: Bundle?) {
val list = partialResults?.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION).orEmpty()
list.firstOrNull()?.let(::handleTranscription)
}
override fun onEvent(eventType: Int, params: Bundle?) {}
}
}

View file

@ -0,0 +1,6 @@
<?xml version="1.0" encoding="utf-8"?>
<adaptive-icon xmlns:android="http://schemas.android.com/apk/res/android">
<background android:drawable="@color/ic_launcher_background" />
<foreground android:drawable="@mipmap/ic_launcher_foreground" />
<monochrome android:drawable="@mipmap/ic_launcher_foreground" />
</adaptive-icon>

View file

@ -0,0 +1,6 @@
<?xml version="1.0" encoding="utf-8"?>
<adaptive-icon xmlns:android="http://schemas.android.com/apk/res/android">
<background android:drawable="@color/ic_launcher_background" />
<foreground android:drawable="@mipmap/ic_launcher_foreground" />
<monochrome android:drawable="@mipmap/ic_launcher_foreground" />
</adaptive-icon>

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.3 KiB

Some files were not shown because too many files have changed in this diff Show more