What is the ORBIT project about?
Novel smartphones apps using Artificial Intelligence (AI) are really useful in making visual information accessible to people who are blind or low vision. For instance, Seeing AI or TapTapSee allow you to take a picture of your surroundings, and then they tell you what things are recognised, for example, “a person sitting on a sofa”. While AI recognises objects in a scene if they are common, at the moment these apps can’t tell you which of the things it recognises is yours, and they don’t know about things that are particularly important to users who are blind or low vision because they are not usually provided to develop apps by sighted people.
Using AI techniques in computer vision to recognise generic objects has made great strides, but it does not work so well for teaching an AI to recognise a specific object that is important to you. For example, you might want to find your set of keys, or the backdoor key rather than the front door key, or maybe you want to identify your white cane. Previous research has started to make some advances to solving the problem by looking at how people who are blind or low vision take pictures and what algorithms could be used to teach AI to recognise specific objects. However, research is currently held back by the lack of available data, particularly from people who are blind or low vision, to use for training and then evaluating AI algorithms for teachable object recognition.
This project, funded by Microsoft AI for Accessibility, has constructed a large dataset by involving blind people. Unlike previous research efforts, our team collected videos since they provide a richer set of information than images and it’s easier for people who are blind or low vision to gather good data. The dataset was collected in two phases: Phase 1 included 48 collectors and 3448 thousands of videos, while Phase 2 gathered data from 52 collectors and contained 1380 videos. Overall we have data for over 500 objects. We anticipate that our dataset might be useful for researchers and developers to implement new algorithms in existing apps or novel wearable systems.
In addition to a dataset, we also developed a curriculum that can teach people who are blind and low vision about how AI works and the importance of data, and how to get involved in developing A.I. themselves.