Background
To build an image recognition AI model, labelers have to annotate images to provide training data. The focus of this project is the GUI for the annotating process, with the resulting AI model used for remote sensing image recognition.
What's our design scope?
What's the status quo?
Block by block annotation for RS image
Block by block
Raw image
The traditional ML annotation process
Export to model training
Manual data labeling
Data cleaning
1
2
Name1
Name1
Name1
Name1
Name2
Name2
Name2
Name2
Training
Iterate
Categorize
Our Focus
Application
Remote sensing image
User
Data Labeler
Tech
AI model
ML model is a BLACKBOX!
Source: SUN Xiao-hua, Review on Human-Intelligent System Collaboration; STUBBS K, HINDS P J, WETTERGREEN D. Autonomy and Common Ground in Human-robot Interaction: a Field Study[J]. IEEE Intelligent Systems, 2007, 22(2): 42-50.
Low Interpretability
Negative impact on collaborative performance and user experience
Low Transparency
Problem1
How?
What if?
Why/Why not?
How does the model make predictions?
If the annotation of these examples were changed, how would the model predict?
Why is this image given such a prediction?
The problem
8hrs
Inefficient; enormous workload
Problem2
Source: User research, see on next page
The process of labeling RS images in traditional machine learning methods is extremely time-consuming, with each RS image requiring up to 8 hours for labeling.
Background
User research & Persona
After conducting 7 interviews with data labelers, here's what I found.
Basic info
22-25
Graduate student in artificial intelligence-related field.
User Goal
Annotate RS images assigned by supervisor
Work environment
Laboratory
Tasks usually take 3 to 4 hours to complete.
Pain
Repetitive work requires a lot of clicking, resulting in low efficiency and physical and mental fatigue.
Insights
Keyboard shortcuts are very important and can greatly improve efficiency.
Opportunity
Opens Google Maps for parallel comparison in ambiguous areas.
Empathize
Increase interpretability
2
Evaluation of the ML model
2
Allow users to experiment
2
Provide adequate transparency
Better annotation
2
3
Adjust weight given to different features; Remove blurry instances
From research to design strategy
操作
Goals
Insights
Emerging research suggests that in some scenarios users may desire richer control over ML systems than simply labeling data.
Different interfaces for different stages of machine learning; Gradual reduction of user paths with ML learning.
Capture intent rather than input.
e.g. Mark instances that the user kept skipping as negative.
Transparency Can Help People Provide Better Labels. System should provide sufficient contextual informations and current predictions.
The average user who doesn't understand the strategy they should take will focus more on labeling than iterating. The interface should promote this strategy through the interaction techniques available and the visual feedback presented.
Provide effective data representations. Make patterns, trends, relationships, outliers and other correlations more observable.
Source：
Power to the People: The Role of Humans in Interactive Machine Learning;
A Review of User Interface Design for Interactive Machine Learning;
Designing for Effective End-User Interaction with Machine Learning;
The Role of Design in Creating Machine-Learning-Enhanced User Experience
1
Efficient
2
Good Model Quality
3
Smooth User Experience
1
Reduce operational steps
1
Plenty of shortcuts for easy label access
1
2
Visually link related elements
3
Increase training/input options
1
3
User-friendly interaction;
Clear visual feedback
3
Caliberated trust
2
3
Beginner's Guide
3
Global navigation -better holistic view
Literature research & solution
Personal project
Feb 2022 - Jun 2022
Interactive machine learning