Picture Smart AI

For a brief introduction to Picture Smart AI™, watch our video on YouTube.

While navigating the digital landscape, you might encounter photos, charts, and other visual elements without an alternative description. This can make it challenging to understand the information provided by the visual content. The JAWS Picture Smart AI feature allows you to submit a photo to be analyzed and then displays a description of the picture in the JAWS Results Viewer window.

This revolutionary update to the original Picture Smart feature, introduced in 2019, integrates with the latest artificial intelligence (AI) technology from services like Chat GPT from Open AI and Claude from Anthropic to offer a whole new level of accessibility to graphical information. Descriptions might offer details such as the number of people in the photo including how they look and what activities they may be performing, recognizable landmarks, landscape details such as the color of the sky, grass, or trees, the layout of information in a visual chart or slide, and much more. Additionally, any text in the picture such as writing on a sign or piece of clothing may also be provided in the description.

To use Picture Smart, press INSERT+SPACEBAR followed by P to access the Picture Smart layer, and then use one of the following keystrokes:

• A – Opens the Picture Smart dialog box where you can acquire an image from a flatbed scanner or the Freedom Scientific PEARL camera and submit it to be described.
• F – Describes the currently selected image in File Explorer. Supported image formats include BMP, JPEG, GIF, TIF, PNG, or HEIC.
• C – Describes any currently selected control. For instance, if focused on  a picture in a Word document or on a social media website like Facebook, this command describes the picture. When using this command on a web page, make sure your browser window is maximized for best results .
• B – Describes the contents of the Windows Clipboard if it contains an image.
• S – Describes the entire screen.
• W – Describes the current application window.
• ENTER – Describes an image based on the current application and context. This eliminates the need to remember multiple keystrokes constantly. For instance:
  • If someone is presenting in Teams, it describes the current slide.
  • If you are on a web page, it describes the current image or control.
  • If you are in File Explorer, it describes the currently selected file.
  • If you are in PowerPoint, it describes the entire screen.
• QUESTION MARK – Displays help for the Picture Smart layer.

After running Picture Smart AI, you will also be alerted to any potential problems with an image that may result in poor descriptions. For example, if your browser window is not maximized, images may be clipped, leading to inaccurate descriptions. In this case, an expandable alert is displayed at the top of the Results Viewer prior to the description. Press ENTER on this alert to view the full details. You can then close the Results Viewer, maximize your browser window, and try running Picture Smart AI on the image again.

Once a submitted image finishes processing, Picture Smart AI displays a short description by default. To display more in-depth descriptions, select the Get more details link near the bottom of the Results Viewer. You can also add SHIFT to a Picture Smart command to immediately display the more detailed descriptions. For example, INSERT+SPACEBAR, P, SHIFT+F.

To demonstrate the difference in the level of detail, here are a few example photos along with resulting descriptions generated using both the original Picture Smart and Picture Smart AI. You can also try this feature out yourself by pressing INSERT+SPACEBAR, P, C while focused on each photo.

Current Picture Smart Sample Description	Picture Smart AI Sample Short Description	Picture Smart AI Sample Long Description
Caption is a group of people sitting on a beach. These tags describe the photo: beach, clothing, girl, group, human face, outdoor, people, person, sky, woman. These tags probably describe the photo: fun, sitting, summer, vacation, water. This tag possibly describes the photo: ground.	The image shows a group of six friends sitting on the beach. They are all smiling and laughing, and some of them are singing. One of them is playing the guitar. The sun is setting in the background, and the waves are crashing on the shore.	The image depicts a group of friends enjoying themselves on a beach at what appears to be sunset due to the warm, golden lighting. There are five individuals in the foreground, with one more partially seen in the background. The group is casually dressed in summer attire and seems to be engaged in a lively and joyful conversation. In the left part of the image, there’s a man sitting on the sand, wearing a light blue button-up shirt and smiling broadly, evidently amused by the conversation. To his right, a woman is laughing heartily, her head tossed back and her hands raised—she’s wearing a yellow top. Centered in the image, another woman is smiling and appears to be sharing a moment of enjoyment with the man on her left; she has her arm extended as if she is pointing at something or someone outside the frame. Turning clockwise, the next woman is in profile and not fully visible, she’s wearing a red and blue checked shirt, and seems engaged with the person beside her. On the right side, another person is mostly turned away from the camera, wearing a white garment that could be a sweater or jacket; this individual appears to be looking back towards the others with a smile. The beach setting provides a tranquil background with the ocean meeting the horizon. The mood is upbeat and the image exudes a sense of friendship, relaxation, and the simple joys of life. There’s a harmonious mix of motion and stillness, with some individuals in mid-gesture, which adds to the dynamic of a candid and spontaneous moment.

Current Picture Smart Sample Description	Picture Smart AI Sample Short Description	Picture Smart AI Sample Long Description
Caption is timeline. These tags describe the photo: display, font, screenshot, text. These tags probably describe the photo: diagram, number.	The image is a timeline that spans from September 2024 to May 2025. It is divided into 5 sections, each with a different task and deadline. The tasks are as follows: – September 2024: Synergize scalable e-commerce – November 2024: Disseminate standardized metrics – January 2025: Coordinate e-business applications – March 2025: Foster holistically superior methodologies – May 2025: Deploy strategic networks with compelling e-business needs	The image you provided is of a slide with a title “TIMELINE” at the top center. Below the title is a horizontal timeline graphic that spans from September 2024 to May 2025. The timeline is divided into five sections, each with a designated month and year, and an associated objective: 1. “SEP 2024” with the objective “Synergize scalable e-commerce” 2. “NOV 2024” with the objective “Disseminate standardized metrics” 3. “JAN 2025” with the objective “Coordinate e-business applications” 4. “MAR 2025” with the objective “Foster holistically superior methodologies” 5. “MAY 2025” with the objective “Deploy strategic networks with compelling e-business needs” Each section is represented by an arrow pointing to the right that starts from the month and extends to the right with the objective contained in the body of the arrow. The colors of the image are mostly shades of blue and grey, with the timeline graphic against a lighter background. At the bottom right corner, there’s a page number or slide number indicated by “11,” and just above it, the year “2024” is written as a note. The overall aesthetic is professional and likely part of a business presentation.

Current Picture Smart Sample Description	Picture Smart AI Sample Short Description	Picture Smart AI Sample Long Description
Caption is chart, pie chart. These tags describe the photo: circle, diagram, screenshot, text. This tag probably describes the photo: font. This tag vaguely describes the photo: design.	The image is a pie chart that shows the favorite fruits of a group of people. There are four categories: melon, apple, mango, and orange. The melon category is blue and takes up 19% of the pie. The apple category is green and takes up 19% of the pie. The mango category is yellow and takes up 25% of the pie. The orange category is orange and takes up 37% of the pie.	The image is a pie chart on a white background with a title at the top that reads “Favorite Fruit.” The chart is divided into four segments, each representing a different fruit with a corresponding percentage, indicating the preference of a group of people. The largest segment is orange, representing oranges, and takes up 37% of the chart. The second largest is yellow, indicating mangoes, accounting for 25%. Two other fruits, melon and apple, each take up 19% of the chart; melon is depicted in blue, and apple in green. Overall, the chart provides a visual representation of the preferred fruits with oranges being the most popular, followed by mangoes, and then melon and apple with equal preference.

AI services

We are continually refining the offerings of Picture Smart AI. We have made significant advancements thanks to the insights gathered during the Early Adopter program, which included assessments of Google’s Gemini and OpenAI’s ChatGPT.

Following further evaluations, we’ve decided to replace Google’s Gemini with Anthropic’s Claude. Claude demonstrated more accurate and reliable image descriptions, with significantly fewer hallucinations. This enhancement is available in the March 2024 release and later.

Relevance of two services

We received great user feedback on the importance of utilizing two services for the image descriptions, mainly because they allow you to spot hallucinations.

You can use the alternative service via the Get more details link at the bottom of the results viewer. In addition, both service descriptions can be displayed directly by pressing the SHIFT key along with a Picture Smart command. For instance, the command INSERT+SPACEBAR, P, SHIFT+W will provide you with two descriptions of the current window.

Read more on our Picture Smart Challenges page.

Asking Questions About a Picture

By popular request, Picture Smart AI offers the ability to send follow-up prompts or questions in order to obtain additional details about a picture that may not have been covered in the initial description.

Once you have performed a Picture Smart AI command such as INSERT+SPACEBAR, P, F, and the description is displayed in the Results Viewer, press ENTER on the Ask questions about this image link near the bottom of the window. If you chose the option to get more details or included SHIFT as part of the command to view detailed descriptions from both AI services, then the link is called Ask Chat GPT questions about this image.

After selecting the link to ask questions, you are immediately placed in an edit box. Type a prompt and press ENTER. For instance, for a picture of a beach at sunset, you might say, “can you describe the sunset and how it looks going down over the ocean.” After a few seconds, during which JAWS will play a sound to indicate progress, the Results Viewer window updates with the response, and JAWS automatically reads it. Your focus remains in the edit box, so you can type another prompt.

In situations where you already have some basic knowledge about an image, you can include ALT with any Picture Smart command to immediately ask a question about specific details.

For example, suppose you were shopping on Amazon for a product like an Multicooker or Air Fryer. You already have a basic idea of what the product is, but you may want to know upfront if the product contains physical buttons or a touch activated display. You could press INSERT+SPACEBAR, P, ALT+C, type your question followed by ENTER, and Picture Smart AI attempts to answer the question about the selected image instead of just providing a general description. Once the results containing the response are displayed, you can still use the Ask questions about this image link to ask follow-up questions if needed.

Your questions and responses are maintained for the current Results Viewer session until you close the window. To review and copy specific responses for pasting in another application, press ESC to exit Forms Mode, and then use H or SHIFT+H to navigate to the heading that corresponds to the question and response you want to copy.

Following is an example picture to demonstrate a question and response.

The image displays an organizational chart with various positions and names. The chart is headed by a President, followed by an Executive Assistant and several Vice Presidents (VPs) overseeing departments such as Finance, Technology, Operations, Marketing, and Production. Under the VPs are different managerial and specialist roles like Accounting, Help Desk, Facilities, Project Manager, Production Manager, and Dispatch. The chart uses color-coding to differentiate between national, domestic, and international roles.

User Question:

Who is marked as international?

Response:

In the organizational chart, the “Transportation VP” role, held by “Ian Hansson,” is color-coded as international. This is indicated by the blue color of the box associated with that position.

Providing Feedback

Even  though Picture Smart AI has transitioned out of the Early Adopter Program and is the new standard for describing pictures, users are still encouraged to provide feedback on there experiences. To provide feedback, use the Give feedback about this feature link at the bottom of the Picture Smart Results Viewer window.

Voice Assistant Commands for Picture Smart

For users of the Voice Assistant feature, you can say “Hey Sharky” along with one of the following suggested voice commands to activate Picture Smart AI.

• “Describe the screen”
• “Describe the current window”
• “Describe the current control”
• “Describe the current file”
• “Describe the picture under the camera”
• “Describe the picture on the clipboard”

Accessing Legacy Picture Smart

In situations where you may need to access legacy Picture Smart functionality, add CTRL to a Picture Smart command. Some users have reported that accessing text in images sometimes works better with the older Picture Smart. We are working on further improving the text recognition of Picture Smart AI.

Mouse Mode (Fusion only)

Mouse Mode in Fusion provides an alternative method to access and utilize Picture Smart AI without relying on JAWS hotkeys. By using Mouse Mode, you can bypass the need for keyboard shortcuts, making it easier to access Picture Smart AI if you prefer mouse navigation or if you encounter issues with hotkeys. This method ensures that the feature is accessible to Fusion users with different preferences and needs.

1. To launch and use Picture Smart AI through Mouse Mode, do one of the following:
   • On the Tools toolbar tab, select the Picture Smart button.
   • Press CAPS LOCK+CTRL+P.
2. After Fusion announces, “Picture Smart Mouse Mode,” click and drag the mouse pointer over the item you want to have described.
3. Once it is processed, the description displays in the Results Viewer window and is also read aloud.

Note: If you press and hold SHIFT just before releasing the mouse button, full AI results from Claude and ChatGPT will display in the Results Viewer window. If you press and hold CTRL just before releasing the mouse button, legacy Picture Smart results will display in the Results Viewer window.

Picture Smart Privacy

All services we use for Picture Smart follow our AI Privacy guidelines. Freedom Scientific, as well as our AI Service partners, do not store or train on the submitted images, prompts, or associated answers. Read more on our AI Privacy Statement page.

Back to JAWS Training