tohoku-nlp/fovit-foit

---
language:
- en
size_categories:
- 100M<n<1B
---

<img src="images/overview.png" width=90%>

## FOVIT

<img src="images/000000125375.jpg" alt="coco 2014 train image (000000125375.jpg)" width=50%>

```json
{
'index': 0,
'id': 125375,
'image': 'images/coco/train2017/000000125375.jpg',
'caption': 'A red, white, and blue train is stationary at a platform in an outdoor train station. The train has multiple carriages, with the front carriage prominently visible. The platform is on the right side of the image, where a group of people is standing near the edge, waiting. Above the platform, there is a red metal canopy providing partial shelter. The station wall behind the platform is a combination of red and white bricks, with a trash can mounted on the wall and a few posters or signs visible. The tracks run horizontally across the bottom of the image, with two sets of rails separated by gravel. The platform on the left side of the image is empty, and the ground is paved with concrete tiles. In the background, there is greenery visible above the station structure, indicating an elevated area with plants and trees.',
'instruction': 'The outline should include the main points of the paper, and at least 15 sections should be highlighted with markdown such as *highlighted section*.',
'question': 'Can you outline the main points of the image, highlighting at least 15 sections with markdown?',
'answer': '1. *Train Description*: A red, white, and blue train is stationary at the platform.\n2. *Train Carriages*: The train has multiple carriages, with the front carriage prominently visible.\n3. *Platform Location*: The platform is on the right side of the image.\n4. *People on Platform*: A group of people is standing near the edge of the platform, waiting.\n5. *Canopy*: Above the platform, there is a red metal canopy providing partial shelter.\n6. *Station Wall*: The station wall behind the platform is a combination of red and white bricks.\n7. *Trash Can*: A trash can is mounted on the station wall.\n8. *Posters/Signs*: A few posters or signs are visible on the station wall.\n9. *Tracks*: The tracks run horizontally across the bottom of the image.\n10. *Rails*: There are two sets of rails separated by gravel.\n11. *Left Platform*: The platform on the left side of the image is empty.\n12. *Ground Paving*: The ground is paved with concrete tiles.\n13. *Background Greenery*: Greenery is visible above the station structure.\n14. *Elevated Area*: The greenery indicates an elevated area with plants and trees.\n15. *Overall Setting*: The scene is set in an outdoor train station.',
'conversations': [
{
'from': 'human',
'value': '<image>\nInstruction: The outline should include the main points of the paper, and at least 15 sections should be highlighted with markdown such as *highlighted section*.\nQuestion: Can you outline the main points of the image, highlighting at least 15 sections with markdown?\nAnswer: '
},
{
'from': 'gpt',
'value': '1. *Train Description*: A red, white, and blue train is stationary at the platform.\n2. *Train Carriages*: The train has multiple carriages, with the front carriage prominently visible.\n3. *Platform Location*: The platform is on the right side of the image.\n4. *People on Platform*: A group of people is standing near the edge of the platform, waiting.\n5. *Canopy*: Above the platform, there is a red metal canopy providing partial shelter.\n6. *Station Wall*: The station wall behind the platform is a combination of red and white bricks.\n7. *Trash Can*: A trash can is mounted on the station wall.\n8. *Posters/Signs*: A few posters or signs are visible on the station wall.\n9. *Tracks*: The tracks run horizontally across the bottom of the image.\n10. *Rails*: There are two sets of rails separated by gravel.\n11. *Left Platform*: The platform on the left side of the image is empty.\n12. *Ground Paving*: The ground is paved with concrete tiles.\n13. *Background Greenery*: Greenery is visible above the station structure.\n14. *Elevated Area*: The greenery indicates an elevated area with plants and trees.\n15. *Overall Setting*: The scene is set in an outdoor train station.'
}
],
'instruction_id': 61,
'model_name': 'gpt-4o-2024-05-13',
'supercategories': ['vehicle', 'person'],
'names': ['person', 'train'],
'category_ids': [1, 7],
'coco_url': 'http://images.cocodataset.org/train2017/000000125375.jpg',
'flickr_url': 'http://farm7.staticflickr.com/6069/6059551045_acb8708522_z.jpg',
'height': 480,
'width': 640,
'license': 1
}
```

## No-FOVIT

<img src="images/000000125375.jpg" alt="coco 2014 train image (000000125375.jpg)" width=50%>

```json
{
'index': 0,
'id': 125375,
'image': 'images/coco/train2017/000000125375.jpg',
'caption': 'A red, white, and blue train is stationary at a platform in an outdoor train station. The train has multiple carriages, with the front carriage prominently visible. The platform is on the right side of the image, where a group of people is standing near the edge, waiting. Above the platform, there is a red metal canopy providing partial shelter. The station wall behind the platform is a combination of red and white bricks, with a trash can mounted on the wall and a few posters or signs visible. The tracks run horizontally across the bottom of the image, with two sets of rails separated by gravel. The platform on the left side of the image is empty, and the ground is paved with concrete tiles. In the background, there is greenery visible above the station structure, indicating an elevated area with plants and trees.',
'question': 'Can you outline the main points of the image, highlighting at least 15 sections with markdown?',
'answer': '1. **Train**: A red, white, and blue train is stationary at the platform.\n2. **Carriages**: The train has multiple carriages, with the front carriage prominently visible.\n3. **Platform (Right)**: The platform is on the right side of the image.\n4. **People**: A group of people is standing near the edge of the platform, waiting.\n5. **Canopy**: A red metal canopy provides partial shelter above the platform.\n6. **Station Wall**: The station wall behind the platform is made of red and white bricks.\n7. **Trash Can**: There is a trash can mounted on the station wall.\n8. **Posters/Signs**: A few posters or signs are visible on the station wall.\n9. **Tracks**: The tracks run horizontally across the bottom of the image.\n10. **Rails**: There are two sets of rails separated by gravel.\n11. **Platform (Left)**: The platform on the left side of the image is empty.\n12. **Ground (Left)**: The ground on the left platform is paved with concrete tiles.\n13. **Greenery**: Greenery is visible above the station structure in the background.\n14. **Plants/Trees**: The elevated area in the background has plants and trees.\n15. **Outdoor Setting**: The train station is in an outdoor setting.',
'conversations': [
{
'from': 'human',
'value': '<image>\nQuestion: Can you outline the main points of the image, highlighting at least 15 sections with markdown?\nAnswer: '
},
{
'from': 'gpt',
'value': '1. **Train**: A red, white, and blue train is stationary at the platform.\n2. **Carriages**: The train has multiple carriages, with the front carriage prominently visible.\n3. **Platform (Right)**: The platform is on the right side of the image.\n4. **People**: A group of people is standing near the edge of the platform, waiting.\n5. **Canopy**: A red metal canopy provides partial shelter above the platform.\n6. **Station Wall**: The station wall behind the platform is made of red and white bricks.\n7. **Trash Can**: There is a trash can mounted on the station wall.\n8. **Posters/Signs**: A few posters or signs are visible on the station wall.\n9. **Tracks**: The tracks run horizontally across the bottom of the image.\n10. **Rails**: There are two sets of rails separated by gravel.\n11. **Platform (Left)**: The platform on the left side of the image is empty.\n12. **Ground (Left)**: The ground on the left platform is paved with concrete tiles.\n13. **Greenery**: Greenery is visible above the station structure in the background.\n14. **Plants/Trees**: The elevated area in the background has plants and trees.\n15. **Outdoor Setting**: The train station is in an outdoor setting.'
}
],
'model_name': 'gpt-4o-2024-05-13',
'supercategories': ['vehicle', 'person'],
'names': ['person', 'train'],
'category_ids': [1, 7],
'coco_url': 'http://images.cocodataset.org/train2017/000000125375.jpg',
'flickr_url': 'http://farm7.staticflickr.com/6069/6059551045_acb8708522_z.jpg',
'height': 480,
'width': 640,
'license': 1
}
```

## FOIT

```json
{
'index': 0,
'id': 125375,
'caption': 'A red, white, and blue train is stationary at a platform in an outdoor train station. The train has multiple carriages, with the front carriage prominently visible. The platform is on the right side of the image, where a group of people is standing near the edge, waiting. Above the platform, there is a red metal canopy providing partial shelter. The station wall behind the platform is a combination of red and white bricks, with a trash can mounted on the wall and a few posters or signs visible. The tracks run horizontally across the bottom of the image, with two sets of rails separated by gravel. The platform on the left side of the image is empty, and the ground is paved with concrete tiles. In the background, there is greenery visible above the station structure, indicating an elevated area with plants and trees.',
'instruction': 'The outline should include the main points of the paper, and at least 15 sections should be highlighted with markdown such as *highlighted section*.',
'question': 'Can you outline the main points of the image, highlighting at least 15 sections with markdown?',
'answer': '1. *Train Description*: A red, white, and blue train is stationary at the platform.\n2. *Train Carriages*: The train has multiple carriages, with the front carriage prominently visible.\n3. *Platform Location*: The platform is on the right side of the image.\n4. *People on Platform*: A group of people is standing near the edge of the platform, waiting.\n5. *Canopy*: Above the platform, there is a red metal canopy providing partial shelter.\n6. *Station Wall*: The station wall behind the platform is a combination of red and white bricks.\n7. *Trash Can*: A trash can is mounted on the station wall.\n8. *Posters/Signs*: A few posters or signs are visible on the station wall.\n9. *Tracks*: The tracks run horizontally across the bottom of the image.\n10. *Rails*: There are two sets of rails separated by gravel.\n11. *Left Platform*: The platform on the left side of the image is empty.\n12. *Ground Paving*: The ground is paved with concrete tiles.\n13. *Background Greenery*: Greenery is visible above the station structure.\n14. *Elevated Area*: The greenery indicates an elevated area with plants and trees.\n15. *Overall Setting*: The scene is set in an outdoor train station.',
'conversations': [
{
'from': 'human',
'value': 'Image Caption: A red, white, and blue train is stationary at a platform in an outdoor train station. The train has multiple carriages, with the front carriage prominently visible. The platform is on the right side of the image, where a group of people is standing near the edge, waiting. Above the platform, there is a red metal canopy providing partial shelter. The station wall behind the platform is a combination of red and white bricks, with a trash can mounted on the wall and a few posters or signs visible. The tracks run horizontally across the bottom of the image, with two sets of rails separated by gravel. The platform on the left side of the image is empty, and the ground is paved with concrete tiles. In the background, there is greenery visible above the station structure, indicating an elevated area with plants and trees.\nInstruction: The outline should include the main points of the paper, and at least 15 sections should be highlighted with markdown such as *highlighted section*.\nQuestion: Can you outline the main points of the image, highlighting at least 15 sections with markdown?\nAnswer: '
},
{
'from': 'gpt',
'value': '1. *Train Description*: A red, white, and blue train is stationary at the platform.\n2. *Train Carriages*: The train has multiple carriages, with the front carriage prominently visible.\n3. *Platform Location*: The platform is on the right side of the image.\n4. *People on Platform*: A group of people is standing near the edge of the platform, waiting.\n5. *Canopy*: Above the platform, there is a red metal canopy providing partial shelter.\n6. *Station Wall*: The station wall behind the platform is a combination of red and white bricks.\n7. *Trash Can*: A trash can is mounted on the station wall.\n8. *Posters/Signs*: A few posters or signs are visible on the station wall.\n9. *Tracks*: The tracks run horizontally across the bottom of the image.\n10. *Rails*: There are two sets of rails separated by gravel.\n11. *Left Platform*: The platform on the left side of the image is empty.\n12. *Ground Paving*: The ground is paved with concrete tiles.\n13. *Background Greenery*: Greenery is visible above the station structure.\n14. *Elevated Area*: The greenery indicates an elevated area with plants and trees.\n15. *Overall Setting*: The scene is set in an outdoor train station.'
}
],
'instruction_id': 61,
'model_name': 'gpt-4o-2024-05-13',
'supercategories': ['vehicle', 'person'],
'names': ['person', 'train'],
'category_ids': [1, 7],
'license': 1
}
```

## No-FOIT

```json
{
'index': 0,
'id': 125375,
'caption': 'A red, white, and blue train is stationary at a platform in an outdoor train station. The train has multiple carriages, with the front carriage prominently visible. The platform is on the right side of the image, where a group of people is standing near the edge, waiting. Above the platform, there is a red metal canopy providing partial shelter. The station wall behind the platform is a combination of red and white bricks, with a trash can mounted on the wall and a few posters or signs visible. The tracks run horizontally across the bottom of the image, with two sets of rails separated by gravel. The platform on the left side of the image is empty, and the ground is paved with concrete tiles. In the background, there is greenery visible above the station structure, indicating an elevated area with plants and trees.',
'question': 'Can you outline the main points of the image, highlighting at least 15 sections with markdown?',
'answer': '1. **Train**: A red, white, and blue train is stationary at the platform.\n2. **Carriages**: The train has multiple carriages, with the front carriage prominently visible.\n3. **Platform (Right)**: The platform is on the right side of the image.\n4. **People**: A group of people is standing near the edge of the platform, waiting.\n5. **Canopy**: A red metal canopy provides partial shelter above the platform.\n6. **Station Wall**: The station wall behind the platform is made of red and white bricks.\n7. **Trash Can**: There is a trash can mounted on the station wall.\n8. **Posters/Signs**: A few posters or signs are visible on the station wall.\n9. **Tracks**: The tracks run horizontally across the bottom of the image.\n10. **Rails**: There are two sets of rails separated by gravel.\n11. **Platform (Left)**: The platform on the left side of the image is empty.\n12. **Ground (Left)**: The ground on the left platform is paved with concrete tiles.\n13. **Greenery**: Greenery is visible above the station structure in the background.\n14. **Plants/Trees**: The elevated area in the background has plants and trees.\n15. **Outdoor Setting**: The train station is in an outdoor setting.',
'conversations': [
{
'from': 'human',
'value': 'Image Caption: A red, white, and blue train is stationary at a platform in an outdoor train station. The train has multiple carriages, with the front carriage prominently visible. The platform is on the right side of the image, where a group of people is standing near the edge, waiting. Above the platform, there is a red metal canopy providing partial shelter. The station wall behind the platform is a combination of red and white bricks, with a trash can mounted on the wall and a few posters or signs visible. The tracks run horizontally across the bottom of the image, with two sets of rails separated by gravel. The platform on the left side of the image is empty, and the ground is paved with concrete tiles. In the background, there is greenery visible above the station structure, indicating an elevated area with plants and trees.\nQuestion: Can you outline the main points of the image, highlighting at least 15 sections with markdown?\nAnswer: '
},
{
'from': 'gpt',
'value': '1. **Train**: A red, white, and blue train is stationary at the platform.\n2. **Carriages**: The train has multiple carriages, with the front carriage prominently visible.\n3. **Platform (Right)**: The platform is on the right side of the image.\n4. **People**: A group of people is standing near the edge of the platform, waiting.\n5. **Canopy**: A red metal canopy provides partial shelter above the platform.\n6. **Station Wall**: The station wall behind the platform is made of red and white bricks.\n7. **Trash Can**: There is a trash can mounted on the station wall.\n8. **Posters/Signs**: A few posters or signs are visible on the station wall.\n9. **Tracks**: The tracks run horizontally across the bottom of the image.\n10. **Rails**: There are two sets of rails separated by gravel.\n11. **Platform (Left)**: The platform on the left side of the image is empty.\n12. **Ground (Left)**: The ground on the left platform is paved with concrete tiles.\n13. **Greenery**: Greenery is visible above the station structure in the background.\n14. **Plants/Trees**: The elevated area in the background has plants and trees.\n15. **Outdoor Setting**: The train station is in an outdoor setting.'
}
],
'model_name': 'gpt-4o-2024-05-13',
'supercategories': ['vehicle', 'person'],
'names': ['person', 'train'],
'category_ids': [1, 7],
'license': 1}
```