AI Content Describer

AI Content Describer gives blind and low vision NVDA users a way to understand visual content that a screen reader can't reach on its own. Point it at a control, an image, the whole screen, or a camera, and it sends the picture to a vision model and reads the description back.

I built it because OCR only solves part of the problem. NVDA is already quite good at pulling words out of a screenshot or scanned page. It can't explain a graph, tell you what is happening in a photo, make sense of an unlabeled game screen, or describe how a window is laid out. Alt text is supposed to cover some of that, but anybody who uses a screen reader knows how often it doesn't.

The add-on fills in those gaps quite nicely. I use it for everything from screenshots and memes to virtual machines, charts, confusing application windows, and the occasional mystery object on my desk.

What you can point it at

You can describe the control with focus, the object under NVDA's review cursor, or everything visible on the screen. It can also work with an image copied to the clipboard, a file selected in Windows Explorer, or a new photo from a connected camera.

Descriptions aren't one shot answers. If the first response leaves something out, ask a follow-up question. You can attach another image to the same conversation, zoom in on a detail, compare two pictures, or ask the model to format complicated results as a table or list.

There is also a local face positioning tool for video calls and recordings. It tells you whether your face is centered in the camera frame without sending the image to an AI service or requiring an API key.

More than image descriptions

Some models can operate the application in front of you with the mouse and keyboard. You describe what you want done, review a consent prompt, then listen as the add-on announces each action before it happens. It pauses when it needs more information, asks again before risky actions, and can be stopped or paused at any time.

This can get you through an inaccessible dialog or a task that depends on clicking something visual. It can also make mistakes. Computer use is still a beta feature, and I wouldn't hand it a sensitive or difficult to reverse job and walk away.

For documents, AI Content Describer supports general vision models as well as models built to understand dense layouts, handwriting, tables, charts, forms, and math. That makes it useful when plain OCR technically finds the words but loses the structure that gives them meaning.

Pick the model that works for you

The add-on includes free access through Pollinations, so you can install it and start describing images without setting up an account or buying API credit. That service is convenient, but it can be unreliable. If you use the add-on regularly, you will probably get better results with your own key from a supported provider.

OpenAI, Google Gemini, Anthropic Claude, Mistral, xAI, Moonshot, vivo, and Datalab models are supported. You can also connect to local or self-hosted models through Ollama, llama.cpp, LiteLLM Proxy, or Seer. Results, prices, and computer control support vary by model, and they change too quickly for me to tell me which one you should be using for a particular task.

Unless you use the local face positioning feature or a local model, remember that the image has to be sent to the provider you selected. The add-on can cache responses to save API usage, and you can tune prompts and response length when the defaults aren't enough.

Getting started

1. Install AI Content Describer from the NVDA add-on store.
2. The first time NVDA starts with it installed, allow it to download the extra libraries it needs.
3. Navigate to AI Content Describer settings under NVDA -> Settings -> AI Content Describer. Please consult project documentation to learn how to generate API keys for the model that you would like to use.
4. Press NVDA+Shift+I to open the main menu and choose what to describe.

NVDA+Shift+U describes the navigator object, while NVDA+Shift+Y describes an image copied to the clipboard. Every gesture can be changed through NVDA's Input Gestures dialog.

The project documentation covers model setup, every command and setting, follow-up conversations, computer use, and the privacy tradeoffs involved in choosing a provider.