Search by meaning, without sending your clipboard anywhere

You rarely remember the exact words of something you copied. You remember that it was the staging database link, or the Wi-Fi password from the café, or the address someone sent last week. Keyword search fails all three: the link says postgres://, the password is in a photo, and the address never contained the word "address".

Clypy's search finds them by meaning, and reads the text in pictures, without sending a single clip to a server. This post explains how it works, including the parts that do not work as well as we would like.

Meaning, as numbers

Search by meaning uses an embedding model: a small neural network that turns a piece of text into a list of numbers, so that texts about similar things end up with similar lists. Clypy runs one on your device, for each clip as it arrives and for your query as you type, and compares the two.

Which model depends on the platform:

• Windows, Linux and Android use all-MiniLM-L6-v2, a widely used open model, in a compressed version of about 23 MB that turns text into 384 numbers. Clypy downloads it once, from a fixed version on Hugging Face, and checks its SHA-256 hash before using it.
• Mac, iPhone and iPad use Apple's sentence embeddings, which are part of the operating system, so there is nothing to download.

The numbers are stored in Clypy's encrypted database, next to the clips, and each device works out its own.

What gets read

Before a clip is embedded, Clypy turns it into one short passage: its type, its title, its first 140 characters, a few derived words such as "tracking number" or "api key", the text in it if it is a picture, and the title and description of a link.

That has a consequence we should be upfront about. A long clip is matched by meaning on its beginning, not its whole body. Paste a 2,000-word document, search for something from its last paragraph, and meaning search will not find it.

Keyword search can. Turn on Search inside clip text in Settings, Privacy & security, and Clypy indexes every word of every clip, inside its encrypted database. It is off by default.

Two searches, one list

Every query runs two searches.

Keyword search goes first, with weights. A match in a title counts three times as much as one in the preview, text found in a picture nearly as much as the preview, and the full body of a clip a little less. It forgives typos and matches abbreviations.

Then the meaning search runs, and its results are merged in. A meaning match has to clear a similarity of 0.35, on a scale where 1 means identical. Below that, the nearest clip is not related to your query; it is just the least unrelated thing in your library, and showing it anyway is how semantic search earns its reputation for making things up. A clip found both ways adds the two scores together and ranks above one found either way alone.

When you know the exact text you want, those guesses get in the way. Exact match only, in the search filters, turns off typo correction, abbreviations and meaning, and matches what you typed.

Every result says why it is there

Search that finds things by meaning can feel like a magic trick, and tricks are hard to trust. So every result says why it is there, in plain words:

• Similar meaning, even though none of your words appear in it
• Matched text in the image: "…"
• Corrected to "…"
• Matched the abbreviation "…"
• Copied from Safari, Captured on your laptop, or Filed under a collection

If a result surprises you, the reason usually explains it. If it does not, that is a bug, and we would like to hear about it.

Reading pictures

The same idea applies to screenshots. When a picture arrives, Clypy reads the text in it with the text recognition each platform already has: Vision on a Mac and iPhone, Windows' own text recognition on a PC, and Google's on-device ML Kit on Android. Linux has no system text recognition, so there Clypy uses ocrs, a small open-source model of about 12 MB that you download when Clypy offers it.

The text it finds is searchable straight away, which is how "wifi password" finds a photo of a café's Wi-Fi card. On a computer the text is also shown on the clip, with a button to copy it. Like the embeddings, it is never synced. Each device reads its own pictures.

Secrets stay out of every index

Anything Clypy marks sensitive is kept out of the keyword index and the meaning index, and its pictures are never read. A concealed API key does not turn up because you searched for the service it belongs to.

What leaves your device

No clip is ever sent to a model, ours or anyone else's. Here is everything in this part of Clypy that touches the network:

• Model downloads. The embedding model on Windows, Linux and Android, and the Linux text model, are downloaded once and checked against a fixed hash. On Android, translation models come through Google's ML Kit.
• Link previews. When you copy a link, Clypy fetches the page's title, description and icon from that site, so the link can be found by what it is about. This is on by default. It skips sensitive, ephemeral and local-only clips, and you can turn it off in Settings, Capture, Link previews.

Nothing else. Embedding, searching and reading pictures all happen on the device in front of you.

Where it falls short

• On Windows and Linux, search by meaning works in English. The model we use there was trained on English text, so other languages fall back to keyword search. On a Mac and iPhone, Apple's embeddings detect the language of each clip and cover several languages.
• Long clips are matched on their beginning. Turn on Search inside clip text if you search inside long documents.
• A new device has catching up to do. A newly paired device works through its library in the background, and until it finishes, older clips are found by keyword only.

All of this is free, on every platform, with no account. If you want to see how it handles your own clipboard, download Clypy and search for something you only half remember.