# Changelog
Source: https://docs.qalti.com/changelog



## Release Notes

Stay up to date with the latest Qalti releases, features, and improvements.

### Version 0.5.7

Read the full release notes: [Qalti Release 0.5.7](https://qalti.com/blog/qalti-release-0-5-7)

### Version 0.5.3

Read the full release notes: [Qalti Release 0.5.3](https://qalti.com/blog/qalti-release-0-5-3)


# Writing Tests - Best Practices
Source: https://docs.qalti.com/getting-started/best-practices



You can write Qalti tests in many different ways, using more general instructions, and they will work. However, based on our experience and feedback from our clients, we've identified practices that make tests more reliable and consistent.

Think of it this way: these guidelines are similar to what you'd follow when outsourcing manual testing. You need clear, detailed documentation for each test case—written for someone who may be seeing your app for the first time. The same principle applies here. Below we break down the specific details to help you write effective Qalti tests.

## Test File Format

A Qalti test is a plain text file with a `.test` extension containing one step per line. We use `//` at the beginning of a line for comments that provide context rather than actionable instructions.

## Start with Context

Begin each test with comment lines explaining what you're testing:

```text theme={null}
// SyncUps is a meeting management app that helps teams organize daily sync-up meetings.
// This test verifies that a past meeting appears in the history list.
```

We recommend using `//` to mark lines as comments. Use comments to provide additional contextual information that helps Qalti Agent understand the app's purpose, test objective, or any other background information. While the Agent reads all lines, comments are typically used for context and guidance rather than actionable test steps.

## Write Clear Actions

Combine the action and expected outcome in a single line:

```text theme={null}
Tap the "Design" sync-up card. The Design sync-up detail screen opens showing "SYNC-UP INFO" section.
```

**Good examples:**

* `Open the SyncUps app. When app loads you will see "Daily Sync-ups" screen with a list of sync-ups.`
* `Tap "Start Meeting" button. The meeting timer screen appears with "Time Elapsed" and "Time Remaining" displayed.`
* `Tap "Save and end". You return to the Design sync-up detail screen.`

**Avoid vague instructions:**

* ❌ `Go to the main screen`
* ✅ `Tap "Daily Sync-ups". The main screen appears with a list of sync-ups.`

## Be Specific

Always use exact element names in quotes and specify what should happen:

* Quote exact text: `"Start Meeting"`, `"Design"`, `"Save and end"`
* Specify locations when needed: `"End meeting" button at the top left`
* Describe visible elements: `"Time Elapsed" and "Time Remaining" displayed`

## Assume First-Time Use

Write as if Qalti is seeing your app for the first time:

* Don't use internal team jargon or abbreviations
* Explain what each screen shows
* State expected outcomes explicitly
* Describe the full context of actions

**Example:**

```text theme={null}
Tap the "End meeting" button at the top left. A dialog appears with the question "End meeting?" and three options: "Save and end", "Discard", and "Resume".
```

## Handle Edge Cases

Account for optional dialogs or permissions:

```text theme={null}
If a speech recognition permission dialog appears, dismiss it by not allowing.
```

## Use Verification Steps

Add explicit verification lines for critical checks:

```text theme={null}
Verify that a new meeting entry with today's date and current time appears in the "PAST MEETINGS" section.
```

Verification steps should:

* State what to check clearly
* Include specific element names or values
* Focus on test objectives

## Complete Test Example

```text theme={null}
// SyncUps is a meeting management app that helps teams organize daily sync-up meetings.
// This test verifies that a past meeting appears in the history list.

Open the SyncUps app. When app loads you will see "Daily Sync-ups" screen with a list of sync-ups.
Tap the "Design" sync-up card. The Design sync-up detail screen opens showing "SYNC-UP INFO" section.
Tap the "Start Meeting" button. The meeting timer screen appears with "Time Elapsed" and "Time Remaining" displayed.
If a speech recognition permission dialog appears, dismiss it by not allowing.
Tap the "End meeting" button at the top left. A dialog appears with the question "End meeting?" and three options: "Save and end", "Discard", and "Resume".
Tap "Save and end". You return to the Design sync-up detail screen. Wait for the screen to update with the new meeting entry.
Verify that a new meeting entry with today's date and current time appears in the "PAST MEETINGS" section.
Swipe the newly created meeting row fully to the left starting from the right edge (near the ">" arrow). The meeting is removed from the "PAST MEETINGS" list.
```

## Using Comments

You can add comments anywhere in your test file using `//` at the beginning of a line:

```text theme={null}
// This is a comment providing context about the app or test
Open the app. The home screen appears.

// The next step handles an optional permission dialog
If a notification permission dialog appears, tap "Allow".

// Now we verify the main functionality
Verify that the dashboard shows today's date.
```

Comments are useful for:

* Explaining the app's domain or purpose at the start of the test
* Describing why a particular test step is important
* Noting edge cases or conditional behavior
* Adding context that helps the Agent make better decisions
* Documenting assumptions or prerequisites

**Tip:** While the Agent reads all lines including comments, we recommend using them primarily for context and background information. This keeps your actionable test steps clear and easy to follow.

## Common Action Patterns

* **Opening apps**: `Open the [App Name] app. The [screen name] appears.`
* **Tapping elements**: `Tap "[Element]". The [expected screen/state] appears.`
* **Input text**: `Input "[text]". [Expected result] appears.`
* **Scrolling**: `Move finger on the middle of the screen down. New content appears below.`
* **Waiting**: `Wait for the screen to update with the new meeting entry.`
* **Verifying**: `Verify that "[element]" [condition].`

## Key Principles

1. **Specificity over brevity** — Better to be clear than concise
2. **Explicit outcomes** — Always state what should happen
3. **No assumptions** — Don't assume Qalti knows your app
4. **One step, one line** — Keep each action on its own line
5. **Test like a human** — Write instructions as you'd explain to a new team member

Following these practices will make your tests more reliable and reproducible.


# CI Quickstart
Source: https://docs.qalti.com/getting-started/ci-quickstart



This guide shows how to run the Qalti CLI in a CI pipeline (GitHub Actions). It includes parallel simulator runs and an example of running a test on a real iOS device.

You can check out the full repository here: [github.com/qalti/qalti](https://github.com/qalti/qalti). It includes source code, CI examples, and demo tests.

Below is how a real CI run looks like under the hood with 4 workers:

![Qalti demo](https://raw.githubusercontent.com/qalti/qalti/main/imgs/ci-4-tests-demo.gif)

## What you can do

* Parallel simulator execution
* Run tests on a real device
* Produce Allure-formatted reports that you can upload to any QA system

## Authentication for CI

Qalti requires your OpenRouter API key for test execution.

Set it in your CI environment:

```bash theme={null}
export OPENROUTER_API_KEY="sk-or-v1-..."
```

Or pass it explicitly with `--token`.

## Install Qalti in CI

Qalti is a macOS app, and the CLI is included in the bundle. You can download and install it in your CI pipeline with this bash script:

```bash theme={null}
curl -L -o Qalti.dmg https://app.qalti.com/releases/Qalti.dmg
hdiutil attach Qalti.dmg -nobrowse -quiet -mountpoint /Volumes/Qalti
cp -R /Volumes/Qalti/Qalti.app /Applications/Qalti.app
hdiutil detach /Volumes/Qalti -quiet

# Run the CLI directly from the app bundle
/Applications/Qalti.app/Contents/MacOS/Qalti cli --help
/Applications/Qalti.app/Contents/Resources/QaltiScheduler --help
```

## Minimal CLI example

This is the easiest way to run your test:

```bash theme={null}
export OPENROUTER_API_KEY="sk-or-v1-..."
/Applications/Qalti.app/Contents/MacOS/Qalti cli change_appearance.test --token "$OPENROUTER_API_KEY"
```

where `change_appearance.test` is a test file with steps in English:

```text theme={null}
Open Settings
Scroll down to find Display & Brightness
Tap it
Change Appearance
Verify Appearance is changed
```

To learn more, run `/Applications/Qalti.app/Contents/MacOS/Qalti cli --help`

<Accordion title="View CLI help output">
  ```text theme={null}
  Qalti CLI - Run iOS tests from command line

  USAGE:
      qalti cli <test-file> --token OPENROUTER_API_KEY [options]

  ARGUMENTS:
      <test-file>              Path to test file (.test, .txt, or .json)

  OPTIONS:
      --token, -t <token>      OpenRouter API key (or set OPENROUTER_API_KEY env var)
      --model <model>           AI model to use (default: gpt-4.1)
                              Available: gpt-5, gpt-5-mini, gpt-5-nano, gpt-4.1,
                              gemini-2.5-pro, claude-4-sonnet, claude-3.5-sonnet
      --prompts-dir <dir>       Custom prompts directory
      --report-path <file>      Output report path (default: ./reports/test_TIMESTAMP.json)
      --allure-dir <dir>        Generate Allure report files in specified directory
      
  DEVICE SELECTION:
      --udid <udid>             Device UDID (takes precedence)
      --device-name <name>      Device name (e.g., "iPhone 16")
      --os <version>            OS version (e.g., "iOS 18.2")
      --type <type>             Device type: simulator (default) or real
      
  OTHER OPTIONS:
      --app-path <path>         App bundle to install before testing (.app or .ipa)
      --iterations <n>          Max test iterations (default: 50)
      --verbose, -v             Enable verbose output
      --help, -h                Show this help

  EXAMPLES:
      # Basic usage
      qalti cli ./tests/login.test --token sk-or-v1-xxx
      
      # Specific device and model
      qalti cli ./tests/login.test --token sk-or-v1-xxx --device-name "iPhone 16" --model claude-4-sonnet
      
      # Using UDID with app install
      qalti cli ./tests/app.test --udid 12345-67890 --app-path ./MyApp.app --verbose
      
      # Custom prompts and report location
      qalti cli ./tests/checkout.test --prompts-dir ./custom-prompts --report-path ./reports/checkout-run.json
  ```
</Accordion>

## Parallel simulator runs (QaltiScheduler)

Use `QaltiScheduler` to execute tests in parallel:

```bash theme={null}
/Applications/Qalti.app/Contents/Resources/QaltiScheduler \
  --tests "./tests" \
  --device-name "iPhone 16" \
  --os "18.3" \
  --workers 4 \
  --model "gpt-4.1" \
  --token "$OPENROUTER_API_KEY" \
  --app-path "./SyncUps-simulator.zip" \
  --allure-dir "./reports/allure"
```

## CI example (GitHub Actions)

This workflow example shows how you can use Qalti in your CI pipelines: [`.github/workflows/qalti.yml`](https://github.com/qalti/qalti/blob/main/.github/workflows/qalti.yml) — it's short, commented, and shows how to:

* Set up Qalti
* Run tests on simulators and on a real device
* Export Allure results and upload artifacts

You can check recent runs in the [Qalti Actions dashboard](https://github.com/qalti/qalti/actions) to see it in action.

## Allure reports

Qalti outputs results in Allure format if you set the `--allure-dir` flag. You can upload them to your QA system of choice. As an example, we use TestOps Cloud and upload via `allurectl`.

![Allure report in TestOps Cloud](https://raw.githubusercontent.com/qalti/qalti/main/imgs/qalti-allure-report-in-testsops.png)

## Contact

* Questions and support: [hi@qalti.com](mailto:hi@qalti.com)
* Qalti website: [qalti.com](https://qalti.com/)


# Your First Test
Source: https://docs.qalti.com/getting-started/first-test



The easiest way to run your first test is on an iOS Simulator with an app build compiled for the simulator. Don't worry—**your app build never leaves your computer**. We don't have access to it. All we receive are screenshots and the test flow itself, similar to test management systems like [qase.io](https://qase.io) or [qameta.io](https://qameta.io).

## Prerequisites

Before running your first test, ensure you have:

* [Qalti installed](/getting-started/install) on your Mac
* Your app bundle available as `.app` built for simulator
* Your OpenRouter API key (from [openrouter.ai/keys](https://openrouter.ai/keys))

If you don't have your own app build available yet, you can [download our demo app (SyncUps)](https://app.qalti.com/SyncUps/SyncUps-simulator.zip). However, we recommend trying Qalti with your own app to see how it works with your specific use case.

## Running Your First Test in Qalti UI

<video />

Follow these steps to run your first test:

1. **Launch Qalti** — Open Qalti from your Applications folder
2. **Add API Key** — Open Settings in Qalti and add your OpenRouter API key
3. **Connect to Simulator** — Connect to an iOS Simulator from within Qalti. Wait a few minutes while the initialization process completes
4. **Install Your App** — Drag and drop your app build (`.app`) into Qalti to install it on the simulator
5. **Write Your Test** — Write your test in plain English. Here's an example test for the SyncUps app:

<Accordion title="View the test">
  ```text theme={null}
  // SyncUps is a meeting management app that helps teams organize daily sync-up meetings.
  // This test verifies that a past meeting appears in the history list.

  Open the SyncUps app. When app loads you will see "Daily Sync-ups" screen with a list of sync-ups.
  Tap the "Design" sync-up card. The Design sync-up detail screen opens showing "SYNC-UP INFO" section.
  Tap the "Start Meeting" button. The meeting timer screen appears with "Time Elapsed" and "Time Remaining" displayed.
  If a speech recognition permission dialog appears, dismiss it by not allowing.
  Tap the "End meeting" button at the top left. A dialog appears with the question "End meeting?" and three options: "Save and end", "Discard", and "Resume".
  Tap "Save and end". You return to the Design sync-up detail screen. Wait for the screen to update with the new meeting entry.
  Verify that a new meeting entry with today's date and current time appears in the "PAST MEETINGS" section.
  Swipe the newly created meeting row fully to the left starting from the right edge (near the ">" arrow). The meeting is removed from the "PAST MEETINGS" list.
  ```

  **Note:** Lines starting with `//` are typically used for comments. We recommend using them to provide context that helps Qalti Agent understand your app and test objectives, while keeping actionable steps clearly separated.
</Accordion>

6. **Choose AI Model** — Qalti allows you to select different LLM models for running tests. We recommend using the **default recommended model** for the best results
7. **Run Your Test** — Click the **Run** button to execute your test. Qalti will interact with your app and provide a detailed report

## Writing Test Steps

Write clear, specific actions as if the system is seeing your app for the first time. Avoid internal team jargon or assumptions about app-specific behavior. Always specify exact button names, element labels, and the expected outcome of each action.

The more specific your instructions, the more reliable and reproducible your tests will be. Without clear guidance, Qalti Agent will make decisions on its own—just like a human tester would—which may lead to unpredictable results.

For a comprehensive guide on writing effective tests, see our [Best Practices](/getting-started/best-practices) page.

## Running Tests via CLI

Qalti also supports running tests from the command line:

```bash theme={null}
export OPENROUTER_API_KEY="sk-or-v1-..."

/Applications/Qalti.app/Contents/MacOS/Qalti \
  cli /absolute/path/to/qalti_tests/your_test.test \
  --token "$OPENROUTER_API_KEY" \
  --app-path /absolute/path/to/YourApp-simulator.zip \
  --device-name "iPhone 16"
```

For more details on CLI usage and CI/CD integration, see the [CI Quickstart](/getting-started/ci-quickstart) guide.


# Install Qalti
Source: https://docs.qalti.com/getting-started/install



## System Requirements

Qalti is currently available for **macOS only**. We support **Apple Silicon (M-series)** processors. While Qalti may work on Intel-based Macs, performance and compatibility are not guaranteed.

To run tests, you need to have **Xcode** and **iOS Simulators** installed on your system. If Xcode or simulators are not found, Qalti will prompt you to install them.

## Installation

You can install Qalti in several ways:

### Option 1: Install via Script

Run this bash script to download and install Qalti automatically:

```bash theme={null}
curl -L -o Qalti.dmg https://app.qalti.com/releases/Qalti.dmg
hdiutil attach Qalti.dmg -nobrowse -quiet -mountpoint /Volumes/Qalti
cp -R /Volumes/Qalti/Qalti.app /Applications/Qalti.app
hdiutil detach /Volumes/Qalti -quiet
```

### Option 2: Download Manually

Download Qalti directly from [qalti.com](https://qalti.com) or using this [direct link](https://app.qalti.com/releases/Qalti.dmg).

## Running Qalti

After installation, you can run Qalti in the following ways:

### UI App

Find and Open Qalti from Applications, or run it from the terminal:

```bash theme={null}
/Applications/Qalti.app/Contents/MacOS/Qalti
```

### CLI

You can run the CLI directly from the app bundle:

```bash theme={null}
/Applications/Qalti.app/Contents/MacOS/Qalti cli --help
/Applications/Qalti.app/Contents/Resources/QaltiScheduler --help
```

To run tests via UI, CLI, or CI, you need your own OpenRouter API key.

Get a key at [openrouter.ai/keys](https://openrouter.ai/keys), then:

* Add it in Qalti Settings for UI runs
* Or set it for terminal/CI runs:

```bash theme={null}
export OPENROUTER_API_KEY="sk-or-v1-..."
```

You can also pass it explicitly in CLI commands:

```bash theme={null}
/Applications/Qalti.app/Contents/MacOS/Qalti cli ./tests/example.test --token "$OPENROUTER_API_KEY"
```


# Qalti — Overview
Source: https://docs.qalti.com/index



<Note>
  Qalti is now open-source under MIT license. The full source code is available on [GitHub](https://github.com/qalti/qalti).
</Note>

Qalti is an AI agent for mobile app testing.
It visually inspects your app and interacts with it the way a user would:
tapping buttons, entering text, scrolling, opening URLs, etc.
You provide a test in plain English; Qalti executes it and reports.

Qalti is a macOS Application with an included CLI. This lets you run tests against local iOS Simulators and connected iPhone/iPad devices with minimal latency. The same binaries run in your environment on macOS (e.g., GitHub Actions on macOS runners or self‑hosted Macs) for CI.

To run tests, add your own OpenRouter API key in Qalti Settings (UI) or pass it via `OPENROUTER_API_KEY` / `--token` for CLI and CI flows.

# How does it work

Qalti Agent behaves like a manual QA engineer: it sees the on‑screen UI like a human and decides what to do next like a human.

Under the hood, Qalti reads the test instructions you provided and runs a tight perception–action loop on your app’s real UI:

* **View** — capture and understand the current screen.
* **Decide** — pick the next best action toward the goal.
* **Act** — execute the tap, type, scroll, or open URL.
* **Verify** — compare the screen with the expected state.
* **Repeat** — iterate until the test flow is done.

Then it produces a run report with a step‑by‑step log and screenshots.

# Qalti Design principles

* **App is the black box** — No source code or UI hierarchy required; Qalti interacts only with the rendered UI. This is why it works for native apps and custom‑rendered UIs (incl. game engines, cross‑platform toolkits).
* **You're in control** — You define the exact flow, inputs, and assertions. The Agent executes deterministically and won't take alternate paths unless explicitly allowed.
* **Common sense, surfaced** — Benign variations are handled (dismiss transient pop‑ups, tolerate minor label changes) to keep runs stable, and every adaptation is highlighted in the report; if semantics would change, the test fails.

# Examples

## RUN App. Training numbers consistency

RUN is a training app that tracks your running workouts.
Its Home‑screen widget shows a snapshot of the latest run with some numbers.
The same metrics are visible inside the RUN app and in Apple’s Fitness app (Sessions).
In this example we verify that all three views stay consistent.

Below is a Qalti test and how the Agent executes it.
**The goal**: verify that the RUN widget shows the same numbers as both the RUN app and Apple’s Fitness app.

<Accordion title="View the test">
  ```text theme={null}
  Find the RUN widget on the Home screen
  Note the numbers displayed to compare in two other places:
  – Distance above the Distance label (km from the last run)
  – Total Time (top left)
  – Heart Rate (top right)
  – Pace (bottom left)
  – Calories burned (bottom right)
  Tap the widget (area with numbers) to open the RUN app
  Scroll down to see the Training
  Verify that Distance, Total Time, Heart Rate, Pace, and Calories match the widget
  Open the Fitness app
  Tap on Sessions
  Verify again that Distance, Total Time, Heart Rate, Pace, and Calories match the widget
  If all 5 numbers match in both apps, mark the test as passed
  ```
</Accordion>

<video />

While running, the Agent highlights the step it is executing.
On the “Scroll down to see the Training” step, it detects that the target content is already visible and skips scrolling.
The Agent compares the numbers across views and notices a 1 BPM difference, so the test is marked as failed. You can pause the video at any moment to read the Agent’s on‑screen reasoning and see why a particular action or comparison was chosen.

This is the **final state** reported by Qalti Agent: In the Fitness app session details, all workout statistics are visible. Every value matches the widget except Heart Rate, which differs by 1 BPM, resulting in a failed run under strict validation criteria.

If a 1 BPM difference is acceptable in your case, add that tolerance to the test, and the Agent will treat the run as passed.

It is easy to run on your app. [Install Qalti](/getting-started/install), then check out our [step-by-step guide with video](/getting-started/first-test) to write and run your first test on the iOS Simulator or a connected device.

# Platform Support

Because it operates at the screen level, apps built with virtually any UI framework are supported. Runs on iOS and iPadOS, on Simulator and real devices. Android will be added soon.

| Platform            | Framework                                    | Simulator | Real device |
| ------------------- | -------------------------------------------- | :-------: | :---------: |
| iPhone / iPad (iOS) | UIKit                                        |     ✅     |      ✅      |
| iPhone / iPad (iOS) | SwiftUI                                      |     ✅     |      ✅      |
| iPhone / iPad (iOS) | React Native                                 |     ✅     |      ✅      |
| iPhone / iPad (iOS) | Flutter                                      |     ✅     |      ✅      |
| iPhone / iPad (iOS) | Unity                                        |     ✅     |      ✅      |
| iPhone / iPad (iOS) | Unreal Engine                                |     ✅     |      ✅      |
| iPhone / iPad (iOS) | WebViews (WKWebView, SFSafariViewController) |     ✅     |      ✅      |
| iPhone / iPad (iOS) | Mobile Browser (Safari, Chrome, etc.)        |     ✅     |      ✅      |
| iPhone / iPad (iOS) | .NET MAUI                                    |     ✅     |      ✅      |
| iPhone / iPad (iOS) | Xamarin.iOS                                  |     ✅     |      ✅      |
| iPhone / iPad (iOS) | Capacitor / Cordova                          |     ✅     |      ✅      |
| iPhone / iPad (iOS) | KMM (Kotlin Multiplatform Mobile)            |     ✅     |      ✅      |
| Android             | —                                            |    WIP    |     WIP     |