Screen awareness

Harness can see the screen you share with it and reason about what is actually there. This is the core of what makes it different from a chat box in a tab.

What you share

Share a tab, a window or your whole screen, using your browser’s normal screen-share picker. If you share a tab with its audio turned on in the picker, Harness can also hear it: it transcribes the tab’s sound on your device, so you can ask about what was said in a call or a video.

Sharing stops on its own after 10 minutes with no activity in the Harness window, unless a watch is running. You can stop sharing any time with one click.

What it feels like

When you share your screen, Harness understands it on your device and answers from what it can see right now. Ask “what is this error,” “what should I click,” or “summarize this page,” and the answer is grounded in the actual pixels in front of you rather than a guess.

Following along

Turn on Follow along in the floating chat to let Harness occasionally contribute while you work. It can connect something on screen to earlier context, flag a discrepancy, or suggest a useful next step. Contributions arrive as ordinary messages; Why this came up shows their context, and Not useful tells Harness to leave that thread alone. Suggested follow-up tasks run only when you select Start.

Interjections follow the persona and writing style you set in Settings → Profile. Changing your profile cancels any pending thought written with the previous settings. A personality can color a useful thought; Harness still stays quiet when there is nothing worth adding.

Follow along uses your selected model and normal inference rates, with at most 12 checks and four contributions per hour. It runs while the chat is open and your screen is shared, yields to your conversation, and drops thoughts that become stale. Turning it off or stopping the share cancels the current consideration. After repeated model failures it pauses; turn it off and on to retry.

The working scratchpad and contribution evidence stay in your browser tab. Selected context and screen evidence travel through your normal inference route for each check, respecting your privacy settings. Only contributions that are actually shown become conversation messages, following your conversation-persistence setting.

Learning what matters to you

Correct Harness naturally: “Price isn’t the issue. For this comparison, I care whether I can move my data out later.” Follow along can retain that scoped criterion and check it when you evaluate another tool. About → Keeping in mind shows what it understood, the words you used, and Forget this. These criteria belong to this conversation; they do not become permanent preferences automatically.

When a later observation applies a criterion, Why this came up connects it to your correction. If you mark that observation That helped, Harness can offer Review personal skill. Review the complete draft, choose a new skill or an existing one to update, and select Save skill or Update skill. Only that approval writes to your on-device skill library. Saved skills become available in future conversations under their model and privacy settings; manage them in Settings → Tools & connections → Skills. Not now dismisses the offer without forgetting the criterion in the current conversation.

In debug mode, About → Considerations shows when Harness checked, stayed quiet, rejected a stale or repeated thought, or hit a model failure. This bounded local view contains decision categories and timings, without screen text or reasoning traces.

Your screen history

Harness keeps a private, on-device timeline of what has been on your screen, so you can ask about the past as well as the present:

• By meaning: “the page about the missing API key.”
• By look: “the green candlestick chart.”
• By exact text: “the tab with HYPE/USDC.”
• By time: “what was I looking at yesterday around 11pm.”
• By activity: “what was I doing before the call.” Harness groups your screen history into episodes of activity, so it can find a stretch of work, not just a single frame.
• By sound: “what did they say about pricing in that video,” from a shared tab’s audio.

Harness finds the right moments in your timeline and shows you the actual frames, then reasons from them. It never invents what a past screen said; if the history does not show what you asked about, it tells you.

How it works

Screen capture, OCR, embeddings, labeling, and history search run in your browser, on WebGPU where available and WASM as a fallback. The raw stream and full history stay local. If the answering model is remote, Harness may send it the selected current or recalled frame or crop needed for that turn. In the anonymous and private privacy settings, a compact, expiring text and audio log of recent changes also passes through Harness for continuity; private e2e keeps that log on your device.

In the anonymous setting, after your device has ranked the best matches for a question about your history, the text read from up to 24 of them goes to Harness’s server for a final relevance check. Private and private e2e keep that check on your device.

Each captured frame moves through a staged pipeline built to be cheap by default and only spend compute when the screen actually changes:

1. Motion gate. A pixel-level difference check discards near-identical frames before any model runs.
2. Scene gate. A CLIP image embedding compares the frame against the last committed keyframe. Only a meaningful change commits a new keyframe, which keeps the timeline dense with signal and sparse with duplicates.
3. Text extraction. On-device OCR uses a PP-OCRv6 detector with a PP-OCRv5 English recognizer, plus an automatic INT8 path for machines without a strong GPU.
4. Labeling. Each keyframe gets a short, searchable label derived from its CLIP scene tags and the OCR text, so the timeline carries a description of every frame without running a heavier model.
5. Embedding. Each keyframe is embedded three ways so it can be retrieved three ways: a CLIP image embedding (visual similarity), a dense-text embedding over the label and OCR (semantic meaning), and a lexical index over the raw OCR (exact strings, domains, tickers).

Retained chronology is indexed through the local sketch index and time segments, so Harness can search by content or reconstruct a sequence around a known time window.

See Models for the specific on-device models this runs on.

Control

You are always in control of what Harness can see. Stop sharing at any time, and Harness simply cannot see your screen until you share again.