How to collect data for the ORBIT dataset project?

Hello and welcome to the ORBIT dataset project! Thank you for joining blind and low vision people across the world to create a large dataset of videos. The first use of this dataset will be to build an app like Seeing A.I. that uses artificial intelligence, abbreviated to A.I., and your phone’s camera to identify things that are important to you, not just generic things. For example, your keys that are no longer where you put them or a friend’s car that is parked somewhere in front of your house.  

To contribute, you will be asked to take several short videos of five things that are important to you. To make sure our algorithms ultimately work for blind and low vision people, it is really important that the videos comes from you and not from sighted helpers. Below are instructions which will help you to take good videos. You can expect to spend about 2 hours reading the instructions and then download the app and start recording the videos.  

Participants who record all their videos at once found it easier to complete their contribution. A donation will be made to a charity of your choice after you submit 5 complete things during our data collection period.  

How does A.I. work?   

Before you begin taking videos, it might help to understand how they will be used. Apps like Seeing A.I. use Artificial Intelligence, or A.I., to automatically recognise the things users record with their phone camera. To do this, these A.I. algorithms have to be “taught” what to look for by analysing videos of the things you want them to recognise – these are called training videos. A.I. algorithms learn best when each training video shows each object on its own from different angles and in different locations. This helps the algorithm learn what is important and what to ignore. Once the algorithm has learned to recognise a thing, we need to test its ability to find that thing in a realistic place and in amongst other things, like keys on a bookshelf. We call this a scene. For us to be able to test the A.I., we need you to record testing videos in these realistic scenes.   

Recording your videos with the ORBIT Camera app

You will collect two kinds of videos for each of your 5 things: training videos for teaching the A.I. algorithm to recognise the thing, and testing videos for testing the performance of the A.I. algorithm. Before you begin using the ORBIT Camera app, it is highly recommended that you complete our tutorial to learn the technique to record your videos. You will be using one of your hands as a reference point to make sure the things you record are in frame because if you point your camera towards your hand there’s a good chance that the thing you record is also captured. 

Step 1. Think of a thing and enter it into the app 

In the ORBIT Camera App home screen, enter the name of your thing and hit the Add thing button. A thing is any item that you regularly need to find or identify. It could be an item that easily gets moved or misplaced, or an item that is difficult or inconvenient to identify by touch alone. If you are stuck for ideas, see the F.A.Q. for a list of things other people have collected. 

Step 2. Record 2 testing videos of the thing 

Step 2.1. Go to the thing

Go to wherever you would normally keep the thing. A testing video must capture the whole scene your thing is part of, including any other objects that are usually there. For example, if your keys are on your desk, the scene is the whole desk.

Step 2.2. Navigate to a new testing video in the app

From your thing’s Record screen, make sure test is selected, and click the Add new video button to bring up the camera controls. 

Step 2.3. Record a testing video of your thing in the whole scene

Place one hand anywhere on your scene, for example, at the edge of the surface, to act as a reference point, and keep it there. Holding the phone in your other hand, bring your phone as close as possible to your reference hand with the camera pointed towards your scene. Use the Record button to start recording the video. Now, draw your phone away from the anchored hand until it reaches your chest. Once you have reached your chest, click the record button again to stop recording. The testing video should last 15 seconds. A tic sound and a vibration will start to play every 5 seconds to help you time your recording. You will hear a double tic sound and a strong vibration when 15 seconds are up. When you reach 20 seconds your recording will automatically stop and your video will be uploaded to the server.

Step 2.4. Repeat from a different position

Face your scene from a different position. Take a second testing video following the guidelines in the previous steps.

Step 3. Record 5 training videos of the thing 

Step 3.1. Take your thing to a new surface with no other objects on it

 You may have to move other things to ensure this. Examples could be a cleared kitchen counter, work desk, or coffee table. Place your thing on the surface, about an arm’s length away from you. This will help you keep it in frame when recording.

Step 3.2. Navigate to a new training video in the app

From your thing’s Record screen, make sure train is selected, and click the Add new video button to bring up the camera controls. 

Step 3.3. Record a training video of your thing

Like with the testing video, place one hand on the surface, very close to your thing. Holding the phone in your other hand, bring your phone as close as possible to your reference hand with the camera pointed towards your scene. Use the Record button to start recording the video. Now, draw your phone away from the anchored hand until it reaches your chest.Then, rotate the thing on the surface so that a different side is facing you. Bring your phone close to the thing again, and then slowly draw it away towards your chest. Repeat this for a third side of the thing, and then click the Record button to stop the recording. In total, a training video should be about 25 seconds. A tic sound and a vibration will start to play every 5 seconds to help you time your recording. You should hear a tic sound and a vibration every time you have moved the phone to your chest. You will hear a double tic sound and a stronger vibration when 25 seconds is up. When you reach 30 seconds your recording will automatically stop and your video will be uploaded to the server.

Step 3.4. Repeat on another 4 surfaces

Move your object to a new clear surface, and take another training video. Repeat this for 4 completely different clear surfaces.

That’s it! Repeat these steps to take testing and training videos for at least 5 things. 

After recording  

When you finish your recordings, the videos will automatically be uploaded to the ORBIT online repository. We will then check your videos to make sure that they don’t have any information in them that could reveal your identity. We will also check if the video you took and the label of that video are appropriate for this research project. Once your videos pass our checks, they will be added to the final dataset which will be made public after the study ends. If they are shown as checked but not as added, you should re-record them. You can delete or re-record any of your videos until the end of the study. 

Download the ORBIT Camera app

Use the following link to download the ORBIT Camera app: <Link>

More information

If you have any questions, difficulties, or concerns about using the ORBIT app to collect data, please email us at info@orbit.city.ac.uk  

We also have more information and tips to share with you on our website: