Skip to content
AIFIM
Research & Practice

accessibility softwarecontrol assistivetech

Click

Software Control for macOS — built for blind and partially sighted users

Status

Click makes programs usable where existing assistive tools fail.

Click is an application you install on your own Mac. It runs alongside the programs you work in and operates them. You say in ordinary language what should happen — “turn the recording down”, “open the equaliser” — instead of specifying where to click.

Purpose

Click investigates whether any software on macOS can be operated entirely without looking at the screen.

The operating system provides ready-made building blocks: buttons, sliders, lists. Such a block tells a screen reader of its own accord what it is and what value it holds. But professional software often draws its interface itself — a mixing desk with a hundred faders, an editing window, a construction grid cannot be built from standard blocks. Whatever is drawn by the program itself remains an image to the operating system, and every assistive tool that relies on the system’s account comes to a halt there.

The gap is therefore not an oversight but a consequence of the architecture — it appeared precisely where software became most capable. The result is harsh all the same: exclusion from professions, from training, from creative work — not because someone lacks the ability, but because a control is an image.

That is exactly where the investigation starts. Click does not depend on a program giving an account of itself. It does not control positions but intents — and an intent can be put into words even by someone who has never seen the screen.

Goal

The goal is professional accessibility for any existing software — not for the programs that are accessible anyway, but precisely for the others. Basic accessibility has existed for thirty years; it stops where the actual work begins.

Four things belong to that: a whole operating system without looking, not a handful of selected programs. Intent-based control everywhere, whatever the application. A companion that describes what happens of its own accord — that a dialogue has appeared, that a process has finished. And accessibility that accumulates: every element once opened up stays open and benefits everyone.

Tasks

The ongoing work tests three assumptions.

  1. Meaning, not wording A sentence in ordinary language is enough, because what counts is not its wording but its meaning. There is no list mapping a particular sentence to a particular button — which is why nobody has to learn a command language. What needs testing is which actions this holds for and where an intent stays ambiguous.
  2. Exact values, verified A self-drawn fader can not only be found but set to a stated value — and afterwards Click checks for itself whether the value was reached. A misclick nobody notices is dangerous; anyone who cannot look for themselves depends on being told. This safety takes precedence over any new feature.
  3. What is learned stays During setup, Click works out how an application is to be controlled and keeps that knowledge. It therefore needs no list of supported programs and waits for no adaptation by the vendor. What needs testing is how stable that stays across program updates.

Testing status

The hardest case is solved. It is hard because three things come together: the control tells the operating system nothing about itself. Finding it is not enough — it has to be set to a particular value. And someone who cannot see cannot check whether that worked.

In a professional music application, Click operates such a fader: you state the value, Click sets it and then verifies the result itself. It has made over eighty controls accessible in that program, almost half of them unlabelled icons. The step beyond has been taken — Click is now learning to operate other applications on macOS as well.

So far it has been tested by its own developers, one of them blind and working in his own field. That is more than usual: assistive tools are mostly built by people who do not need them. It is also a starting point that cannot see its own blind spots. Other people, other ways of working, other degrees of sight, other programs: that testing begins now. A letter of intent is in place for working with an advocacy organisation for blind and partially sighted people.

Still to be added

The name of the organisation, once it has cleared us to name it — and whether the umbrella body or a regional organisation is named.

One thing is explicitly still outstanding. Today Click takes a moment to find a control. A speed-up has been built and measured, but not yet enabled — it first has to prove itself in continuous use.

What happens to the data

Anyone using a screen reader gives that tool complete access — not occasionally but always. Banking, medical letters, private messages. People who see can check for themselves if in doubt; people who do not see have to be able to rely on the tool.

That is why Click works entirely on your own machine. What you type or say, and what is on your screen, is not transmitted to any third-party service. For spoken input, Click uses the speech recognition built into macOS.

And the work uses the publicly documented interfaces of the operating system — the paths intended for assistive tools. No third-party program is interfered with, and nothing in them is altered.

An offer to software makers

Click is not being built against the vendors. Others laid the foundation: with its screen reader and its accessibility interface, Apple created an infrastructure without which this project would not be possible. What we add begins where the standard building blocks end.

What works from the outside works more easily from the inside. Anyone who has the source code of their own program can achieve the same accessibility with far less effort — they only need to know where to start. We are glad to show exactly that. The technology behind it is published under a free licence — to adopt, to adapt, to build in.

Who works on it

Click is being built within the association. It is led by Petar Nikolic, who formulated the research question out of his own experience; he has developed it with Dejan Nikolic from the first hour.

The question comes out of the work on PapoNox, which showed that a professional music application can be operated without sight. Click asks the next question — whether the same holds for any software at all.

Who it is built for

For blind and partially sighted people, especially those who want to work with graphical professional software in their job or their art. For employers and training institutions that want to open up workplaces where such software is used. For people with lasting motor impairments who work without a mouse.

And for everyone else: what is a basic requirement for blind users is a gain in reliability for sighted ones. You trigger recurring sequences by voice, hit a value exactly instead of dragging towards it, and can keep working when your hands are not free.

More about the project