In our research, we improved two existing datasets for tracking multiple objects (GMOT-40 and AnimalTrack) by adding text descriptions. These enhanced datasets are named 'Refer-GMOT40' and 'Refer-Animal'.

'Refer-GMOT40' includes 40 videos covering 10 different types of real-world objects, with each type having 4 video sequences. 'Refer-Animal' contains 26 videos focusing on 10 common types of animals.

Each video in these datasets has been carefully annotated with several details:

For text label:

• class_name: The general category of objects being tracked.
• class_synonyms: Other names or terms for the class.
• definition: A description of the objects being tracked.
• include_attributes: Characteristics of the tracked objects based on what can be seen.
• exclude_attributes: Characteristics that identify objects within the same category that are not being tracked.
• caption: Descriptions of the objects being tracked. For tracking all objects in a class, the caption is in the format: "Track [visible attributes] [class name]". When tracking a specific subset, the format is: "Track [visible attributes] [class name] while excluding [untracked attributes] [class name]".
• track_path: The exact tracking path is stored separately, following the standard format for multiple object tracking challenges.

For track label:

each line will contain 9 elements, seperated by commas

<frame>, <id>, <bb_left>, <bb_top>, <bb_width>, <bb_height>, <conf>, <x>, <y>

• frame: index of frame in video sequence
• id: id of object accord to tracker
• bb_left: x coordinate for top left
• bb_top: y coordinate for top left
• bb_width: width of the box that contains object
• bb_height: height of the box that contains object
• conf: confidence score but get 1 as default
• x: get 1 as default
• y: get 1 as default

The annotations are formatted in JSON, and we provide examples to illustrate how they are structured. This data, prepared by 4 annotators, will be shared publicly.

Text label for referring with specific attributes
{
    video: "",
    label:{
        class_name: "",
        class_synonyms:[],
        definition: "",
        include_attributes: []
        exclude_attributes: []
        caption: "",
        track_path: "",
    }
}
                    

Track label for associating objects' IDs through time
1, 1, xl, yt, w, h, 1, 1, 1
1, 2, xl, yt, w, h, 1, 1, 1
1, 3, xl, yt, w, h, 1, 1, 1
2, 1, xl, yt, w, h, 1, 1, 1
2, 2, xl, yt, w, h, 1, 1, 1
2, 3, xl, yt, w, h, 1, 1, 1
3, 1, xl, yt, w, h, 1, 1, 1
3, 2, xl, yt, w, h, 1, 1, 1
3, 3, xl, yt, w, h, 1, 1, 1