Data Collection – The ORBIT (Object Recognition for Blind Image Training) Dataset

We are no longer collecting any data for this project. You can read about our experiences with the data collection in our forthcoming publication at ACM ASSETS 2021.

Here’s the information we gave for collecting data for Phase 2.

Help us build better AI systems for people who are blind or low vision!

Welcome to the ORBIT dataset project! Thank you for joining blind and low vision people across the world to create a large dataset of videos to develop better AI for your community. We are taking a disability-first approach to building our AI algorithms rather than adapting mainstream algorithms to people who are blind or low vision. To do this, we need data that reflects the people who are going to use it. Join us in creating an AI system that can recognize things that matter to you, from your keys that someone keeps moving to the door of a new friend’s house that you haven’t travelled to before.

In this page you will find all the information needed to take part in the project. Before you download the app and start recording your videos, you should read the information we provide very carefully. You will also need to read Section 2: filming instructions in detail.

Section 1: A summary of the data collection project.
Section 2: Filming instructions on how to record your things
Section 3: A tutorial with a narrated demonstration of the filming techniques
Section 4: Frequently Asked Questions (FAQs)
Section 5: How to navigate in the ORBIT Camera app
Section 6: Download link for the ORBIT Camera app
Section 7: Online training sessions

Section 1: Project summary

The ORBIT project invites you to record videos of things that are important to you. These videos will be added to an open dataset to help create new Artificial Intelligence technologies for blind and low vision users around the world. We want to make personalised object recognisers that work for you! 

During the data collection, you will use an iPhone app to film short videos of at least 5 things you might like your phone to recognise. For each thing you will need to take 7 videos using different filming techniques and in different locations. More details about the types of videos and the filming techniques can be found in the filming instructions.

We anticipate that collecting your videos will take no longer than 1 hour. However, you will be able to take videos of your things using the app at any time until the 29th of November 2020. After taking a video, it will be automatically uploaded to the ORBIT server for checking. If the video doesn’t include any personal identifying information and is considered appropriate, it will be added to the open dataset. We will use push notifications to inform you about your progress and keep you posted about the project.

When you complete your videos, a £25 donation will be offered to a charity that supports blind and low vision communities, on your behalf. The more people participate in the data collection, the higher the donations!

Section 2: Filming Instructions

The first use of this dataset will be to build an app like Seeing AI that uses artificial intelligence, abbreviated to AI, and your phone’s camera to identify things that are important to you, not just generic things. For example, your keys that are no longer where you put them or a friend’s car that is parked somewhere in front of your house.

To contribute, you will be asked to take several short videos of five things that are important to you. To make sure our algorithms ultimately work for blind and low vision people, it is really important that the videos comes from you and not from sighted helpers. Below are instructions which will help you to take good videos. You can expect to spend about 1 hour reading the instructions and then downloading the app and recording the videos.

Participants who record all their videos at once found it easier to complete their contribution.

How does AI work?

Before you begin taking videos, it might help to understand how they will be used. Apps like Seeing AI use Artificial Intelligence, or AI, to automatically recognise the things users record with their phone camera. To do this, these AI algorithms have to be “taught” what to look for by analysing videos of the things you want them to recognise – these are called training videos. AI algorithms learn best when each training video shows each thing on its own from different angles and in different locations. This helps the algorithm learn what is important and what to ignore. Once the algorithm has learned to recognise a thing, we need to test its ability to find that thing in a realistic place and in amongst other things, like keys on a bookshelf. We call this a scene. For us to be able to test the AI, we need you to record testing videos in these realistic scenes.

Recording your videos with the ORBIT Camera app

You will collect two kinds of videos for each of your 5 things: training videos for teaching the AI. algorithm to recognise the thing, and testing videos for testing the performance of the AI algorithm. Before you begin using the ORBIT Camera app, it is highly recommended that you complete our tutorial to learn the technique to record your videos. You will be using one of your hands as a reference point to make sure the things you record are in frame because if you point your camera towards your hand there’s a good chance that the thing you record is also captured.

Step 1. Think of a thing and enter it into the app

In the ORBIT Camera App home screen, enter the name of your thing and hit the Add thing button. A thing is any item that you regularly need to find or identify. It could be an item that easily gets moved or misplaced, or an item that is difficult or inconvenient to identify by touch alone. If you are stuck for ideas, see the FAQ for a list of things other people have collected.

Step 2. Record 2 testing videos of the thing

Step 2.1. Go to the thing

Go to wherever you would normally keep the thing. Capture a testing video of your thing and all the things around it – the whole scene. It is important that this video includes other things that are usually there. For example, if your keys are on your desk, the scene is the whole desk and includes lamp, books etc.

Step 2.2. Navigate to a new testing video in the app

From your Things list screen, either enter a name for your thing or select your thing from the list. This will bring up the Thing record and review screen. Make sure you’re in the correct test video slot by swiping up or down in the recording slots. You are now ready to record a testing video!

Step 2.3. Record a testing video of your thing in the whole scene

Place one hand anywhere on your scene, for example, at the edge of the surface, to act as a reference point, and keep it there. Holding the phone in your other hand, bring your phone as close as possible to your reference hand with the camera pointed towards your scene. Use the Record button to start recording the video. Now, draw your phone away from the anchored hand until it reaches your chest. Once you have reached your chest, click the record button again to stop recording. The testing video should last 15 seconds. A tic sound and a vibration will play every 5 seconds and a double tic sound and a strong vibration will play when 15 seconds are up. After this, your recording will automatically stop and your video will be uploaded to the server.

Step 2.4. Repeat from a different position.

Face your scene from a different position. Take a second testing video following the guidelines in the previous steps.

Step 3. Record 5 training videos of the thing

Step 3.1. Take your thing to a new surface with no other things on it.

You may have to move other things to ensure this. Examples could be a cleared kitchen counter, work desk, or coffee table. Place your thing on the surface, about an arm’s length away from you. This will help you keep it in frame when recording.

Step 3.2. Navigate to a new training video in the app.

From your Things list screen, either enter a name for your thing or select your thing from the list. This will bring up the Thing record and review screen. Make sure you’re in the correct train video slot by swiping up or down in the recording slots. You are now ready to record a training video!

Step 3.3. Record a training video of your thing

Like with the testing video, place one hand on the surface, very close to your thing. Holding the phone in your other hand, bring it as close as possible to your reference hand with the camera pointed towards your scene. Use the Record button to start recording the video. Now, draw your phone away from the anchored hand until it reaches your chest. Then, rotate the thing on the surface so that a different side is facing you. Bring your phone close to the thing again, and then slowly draw it away towards your chest. Repeat this for 2 more sides of the thing, and then click the Record button to stop the recording. In total, a training video should be about 25 seconds. A tic sound and a vibration will play every 5 seconds to help you time your recording. You should hear a tic sound and a vibration every time you have moved the phone to your chest. You will hear a double tic sound and a stronger vibration when 25 seconds is up. When you reach 30 seconds your recording will automatically stop and your video will be uploaded to the server.

Step 3.4. Repeat on another 4 surfaces

Move your thing to a new clear surface, and take another training video. Repeat this for 4 completely different clear surfaces.

Step 4. Repeat these steps to take testing and training videos for at least 5 things.

After recording

When you finish your recordings, the videos will automatically be uploaded to the ORBIT online repository. We will then check your videos and the thing names to make sure that they don’t have any information in them that could reveal your identity or contain anything explicit or offensive. Once your videos pass our checks, they will be added to the final dataset which will be made public after the study ends. Going through each thing you’ve created in the main thing list screen tells you whether you have completed all videos for that thing, and whether any videos are still waiting to be checked by the research team. If a video includes any personal identified information or the thing is not visible in the video, you should receive a notification to re-record your video. You can delete or re-record any of your videos until the end of the study. When all your videos are completed and checked, you will receive a notification to delete the videos and the app.

Section 3: Tutorial

In this tutorial, we will explain how to record your videos using the ORBIT Camera App. Below you will find 2 tutorials with narrated demonstrations where we’ll practice how to take a testing and a training video, with a mug as an example. This tutorial will take about 10 minutes to complete, and you can use it to practice as many times as you like with any other object.

Before we start, let’s first get set up. You will need your mobile phone with the ORBIT Camera App installed, a mug or any other thing of your preference, and a surface to place it.

So that you can practice with the ORBIT Camera App while following this tutorial, it will be easiest if you play the tutorial on another device, for example a laptop or iPad.

You can pause the videos while you get yourself ready or if you want more time to do the filming.

Let’s get started!

Video 1: How to record a testing video?

Video 2: How to record a training video?

Well done on completing this tutorial! You can re-watch it at any time to practice how to take testing and training videos of your things using the ORBIT Camera App.

Thank you for your contribution to the ORBIT Dataset Project!

Section 4: Frequently Asked Questions

How many things should I film?

You must film at least 5 different things.

How many ‘testing’ and how many ‘training’ videos I should take?

For each of your selected things, you must take at least 2 testing videos and 5 training videos. Without the testing videos, we won’t be able to evaluate how good our AI is, so please make sure you include them.

What are the main differences between a ‘training’ and a ‘testing’ video?

There are 2 main differences between ‘training’ and ‘testing’ videos:

Testing videos should include other things, whereas training videos should be on a clear surface without any other things being recorded. It is important that testing videos contain other objects to test if the AI algorithm is identifying the correct object.

In a testing video we only want you to record one side of the thing, while in a training video we want you to rotate the thing so you show us four sides of it.

Is it okay to reuse locations for recording videos?

It is okay if you reuse locations and record a training video of one thing on the same surface as the training video of another thing. Please do not record two training videos for the same thing in the same location.

I’m stuck thinking of a thing to record. Any suggestions?

In a previous data collection, many people chose their keys, cane, wallet, headphones, mugs and sunglasses. You can also include larger things that you cannot move around, like cars, rubbish bins or doors, but you will have to adjust the filming techniques a bit. You can also think of things that might be similar but difficult to distinguish by touch, like socks, backpacks, dog leads, earphones or face masks.

What filming technique should I use to record videos of larger or immovable things?

If your thing is large or immovable, you might not be able to rotate it or move it to different places to record your training videos. You will instead take all its training videos in one place, but rather than rotating your thing around on a surface, you will move yourself around your large or immovable thing. You will record 5 videos of your thing, each taken from a different position or angle facing your thing.

For taking a testing video of a large or immovable thing, you should follow the same instructions as for your small thing. Again here, make sure that you capture the whole scene your thing is part of. It is important that this includes other objects that are usually there beside your thing. For example, if you are filming your door maybe other things are around such as shoes, coats or umbrellas.

Are there any ways to speed up recording the videos?

Currently, the app takes you through recording the different kinds of videos for each thing in different locations. However, you might be quicker by going around 5 different locations and then recording training videos for each of your 5 things. Just remember to make sure to have the right thing selected!

Do I need to have a minimum iOS version to download the app?

Yes, you need to have an iPhone with iOS 13.2 or higher in order to download the ORBIT Camera app.

How much space do I need on my iPhone to record all the videos?

You will need around 0.5GB of free storage space on your phone to record all videos. You can check how much space you have on your phone at Settings – General – iPhone Storage. After the study has finished, you can delete the app and reclaim this space. We will also free up space on your phone if you run low by uploading the video, and then deleting it from your phone.

Some of my videos have not been uploaded yet. What should I do?

We upload the videos on the background when you are on WiFi and plugged in to charge the phone. Make sure you leave your phone charging and on Wifi overnight while you are sleeping and the videos should upload.

What is going to happen if I stop using the ORBIT Camera app before recording all videos per object?

You are free to stop using the app at any time during the study and without any reason. However, in order to contribute to the project and for us to make a donation to a charity on your behalf, you will need to take all 35 videos – 5 things, 7 videos each.

How can I take good videos without knowing if my object is in frame?

We understand that in some cases you won’t get the object framed exactly in the middle of the camera’s view. Your videos do not need to be perfect. We have tried to make the user instructions detailed enough so you can take the videos by yourself. You might have to read the instructions a few times to get the hang of it.

How do I make sure that my hand does not obscure the object?

Try to use your hand as an anchor point, to help you point the camera towards the object. Depending on the size of the object and the filming technique you follow, we recommend placing you hand directly next to your thing.

Should I get help from a sighted person to help me film the videos?

Help from a sighed person is not required. We are building technologies for blind users, so the ORBIT dataset should include videos of things collected by you!

Can I delete the videos from my phone at the end of the study?

Once the videos are published in the dataset, you can delete the whole app – and the videos with it. This will happen at the end of the study period. We’ll send a notification to say so, and the published status is shown in the app. In the meantime, if you run low on storage space, iOS will make space by removing videos that have successfully uploaded. If this happens, you will see our placeholder video – a spinning ORBIT cup – instead of the video you took.

You should only delete individual videos from the app during the study if you want to remove it from the dataset contribution you’re making.

I have a problem installing or using the app. Where can I get help?

For any problems installing or using the app please email us at info@orbit.city.ac.uk

There’s a technical problem with the app. Where can I get help?

For any technical problems with the app please email us at info@orbit.city.ac.uk

Section 5: The ORBIT Camera app

The app has two main screens. The first is called the Things list screen and is a list of the things you have chosen to video. If you select one of your things, you will be taken to a second screen called the Thing record and review screen. This is where you can record videos of that thing.

Your Things list will be empty on your first use. To add something, select the Thing name text field and type a name. When you click “Go” or the “Add thing” button, you will be taken to the record and review screen. On this screen you will find seven recording slots to fill. Here is how the screen is structured:

Recording slot selector. The first two slots are for testing videos, the next five are for training videos. Swipe up or down to cycle through the recording slots.

Camera controls. If the selected slot does not yet have a video recorded, the camera controls will be brought up on screen. The camera controls element gives the filming tip for the video type, and contains the record button, and a ‘next recording slot’ button. Activate the record button to start and stop recording. If you wish to re-record straight away, activate the record button again – the camera controls are not dismissed until you leave the slot. Activate the next recording slot button to cycle through available recording slots.

If the selected slot has a video, the rest of the screen gives you information about that video:

Video recorded date.

ORBIT dataset status. Once the app has uploaded the video to our online repository, this status will say “uploaded” The app needs power and Wi-Fi to upload your videos.

ORBIT dataset status: Once we have checked that your video is appropriate, this status will say “checked”.

ORBIT dataset status. Once the ORBIT dataset has been created and published, this status will say “published”. Once published, any further changes in your app will not be registered.

Delete video button. Removes this video from your thing’s video collection, allowing you to re-record. If the video has not yet been published, this will delete the video from your device and our online repository. Once published, this will only delete the video from your device.

You can freely navigate between the Things list screen and the Thing record and review screen for each of your things. This means you can add things, and videos of any of those things, as opportunities arise. You do not have to add all the videos for a thing in one go.

Each screen has a button that brings up a sheet with more information. The Things list screen has an ORBIT instructions button that brings up a sheet with this text. The Thing record and review screen has a Recording instructions button that brings up the relevant recording instructions.

Section 6: Link to download the ORBIT Camera app

To download the ORBIT Camera use the following link: https://apps.apple.com/app/id1496321313

Section 7: Online training sessions

We understand that taking all these videos is not easy so we are now offering live online training sessions through Zoom to teach you how. The sessions will be on the dates and times listed below. Please register on the session of your preference by sending us an email using the links below. You will then receive the Zoom link to join the session.

During the session, we will:

– give you more background on the project and what the benefits are to blind and low vision people around the world

– describe how to select and arrange the things that you film

– take you through the steps to record videos of the things that you selected

– record a video together of one of the things you select

– be on hand to help you during the rest of the session while you record videos by yourself

– answer any other questions you have or help you if things do wrong