RSVQA low resolution#

This page describes the usage of Dataloader and Datamodule for the low resolution version of RSVQA, a VQA dataset based on Sentinel-2 images over the Netherlands. It was first published by Lobry et al. [4]. The dataset can be found on zenodo . A small example of the data used is distributed with this package.

This module contains two classes, a standard torch.util.data.Dataset and a pytorch_lightning.LightningDataModule that encapsulates the Dataset for easy use in pytorch_lightning applications. Questions and Answers are read using JSON files.

RSVQALRDataSet#

In its most basic form, the Dataset only needs the base path to the image and json files. The path should follow the same structure as it is when downloaded from the official zenodo page and extracted using the official website with images, questions and answer next to each other. The official naming for files is expected.

The full data path structure expected is

datapath = {
    "images": "/path/to/Images_LR",
    "train_data": "/path/to/jsons",
    "val_data": "/path/to/jsons",
    "test_data": "/path/to/jsons"
}

Note, that the keys have to match exactly while the paths can be selected freely.

from configilm import util
util.MESSAGE_LEVEL = util.MessageLevel.INFO  # use INFO to see all messages

from configilm.extra.DataSets import RSVQALR_DataSet
 
ds = RSVQALR_DataSet.RSVQALRDataSet(
    data_dirs=my_data_path  # path to dataset
)

img, question, answer = ds[4]
img = img[:3] # only choose RGB channels

Size: torch.Size([3, 256, 256])
Question: are there less buildings than grass areas?
Question (start): [101, 2024, 2045, 2625, 3121, 2084, 5568, 2752, 1029, 102, 0, 0, 0, 0, 0]
Answer: no
Answer (start): tensor([0., 0., 0., 0., 0., 0., 0., 0., 1.])

../../_images/286302871120a44ccee48f35ebdb1bfaf59e62e4d40b1652093c50e0a70ee532.png

Tokenizer and Tokenization#

As we can see, this Dataset uses a tokenizer to generate the Question out of a natural language text. If no tokenizer is provided, a default one will be used, however this may lead to bad performance if not accounted for. The tokenizer can be configured as input parameter.

from configilm.ConfigILM import _get_hf_model

tokenizer, _ = _get_hf_model("prajjwal1/bert-tiny")

ds = RSVQALR_DataSet.RSVQALRDataSet(
    data_dirs=my_data_path,  # path to dataset
    tokenizer=tokenizer
)
img, question, answer = ds[0]

Tip

Usually this tokenizer is provided by the model itself as shown in the VQA example during dataset creation.

During tokenization a sequence of tokens (integers) of specific length is generated. The length of this sequence can be set with the parameter seq_length. If the generated tokens are shorter than the sequence length, the sequence will be padded with zeros. If it is longer, the sequence is truncated.

Note

Most tokenizer use an ‘End of Sequence’ token that will always be the last one in the non-padded sequence.

ds = RSVQALR_DataSet.RSVQALRDataSet(
    data_dirs=my_data_path,  # path to dataset
    tokenizer=tokenizer,
    seq_length=16
)
_, question1, _ = ds[0]
print(question1)

[101, 2024, 2045, 2625, 3121, 2084, 8206, 5568, 2752, 1029, 102, 0, 0, 0, 0, 0]

ds = RSVQALR_DataSet.RSVQALRDataSet(
    data_dirs=my_data_path,  # path to dataset
    tokenizer=tokenizer,
    seq_length=8
)
_, question2, _ = ds[0]
print(question2)

[101, 2024, 2045, 2625, 3121, 2084, 8206, 102]

The tokenizer can also be used to reconstruct the input/question from the IDs including the special tokens:

print(f"Question 1: '{tokenizer.decode(question1)}'")
print(f"Question 2: '{tokenizer.decode(question2)}'")

Question 1: '[CLS] are there less buildings than circular grass areas? [SEP] [PAD] [PAD] [PAD] [PAD] [PAD]'
Question 2: '[CLS] are there less buildings than circular [SEP]'

or without:

print(f"Question 1: '{tokenizer.decode(question1, skip_special_tokens=True)}'")
print(f"Question 2: '{tokenizer.decode(question2, skip_special_tokens=True)}'")

Question 1: 'are there less buildings than circular grass areas?'
Question 2: 'are there less buildings than circular'

Selecting Bands#

Like for the BigEarthNet v1.0 DataSet, this DataSet supports different Band combinations. Currently, the selection is limited to some preconfigured combinations. Which bands are used is defined by the number of channels set in the Dataset. The selection can be seen when we use a faulty configuration.

try:
    ds = RSVQALR_DataSet.RSVQALRDataSet(
        data_dirs=my_data_path,  # path to dataset
        img_size=(-1, 120, 120)
    )
except AssertionError as a:
    print(a)

Splits#

It is possible to load only a specific split ('train', 'val' or 'test') in the dataset. The images loaded are specified using the json files in the specified path. By default (None), all three are loaded into the same Dataset.

_ = RSVQALR_DataSet.RSVQALRDataSet(
    data_dirs=my_data_path,  # path to dataset
    split="test",
    tokenizer=tokenizer
)

Restricting the number of loaded images#

It is also possible to restrict the number of images indexed. By setting max_len = n only the first n images (in alphabetical order based on their filename) will be loaded. A max_len of None, -1 or larger than the number of images in the json file(s) equals to load-all-images behaviour. The number of images loaded will also change the answer space generated during training splits.

_ = RSVQALR_DataSet.RSVQALRDataSet(
    data_dirs=my_data_path,  # path to dataset
    max_len=10,
    tokenizer=tokenizer
)

_ = RSVQALR_DataSet.RSVQALRDataSet(
    data_dirs=my_data_path,  # path to dataset
    max_len=100,
    tokenizer=tokenizer
)

Select Number of Classes or specific Answers#

For some applications, it is relevant to have only a certain number of classes as valid output. To prevent a dimension explosion if there are too many possible classes, the number of classes can be limited. For the ‘train’ split, it is then automatically determined which combination of classes results in the highest reduction of the dataset.

train_ds = RSVQALR_DataSet.RSVQALRDataSet(
    data_dirs=my_data_path,  # path to dataset
    split="train",
    tokenizer=tokenizer,
    num_classes=3
)

These selected answers can be re-used in other splits or limited if only a subset is required.

Note

The number of classes does not necessarily match the number of answers. If there are fewer answers then classes, the last classes will never be encoded in the one-hot encoded answer vector. If there are more, an IndexError will happen during accessing a non encode-able element.

print(f"Train DS: {train_ds.answers}")

ds = RSVQALR_DataSet.RSVQALRDataSet(
    data_dirs=my_data_path,  # path to dataset
    split="val",
    tokenizer=tokenizer,
    selected_answers=train_ds.answers
)
print(f"Val DS 1: {ds.answers}")

ds = RSVQALR_DataSet.RSVQALRDataSet(
    data_dirs=my_data_path,  # path to dataset
    split="val",
    tokenizer=tokenizer,
    selected_answers=train_ds.answers[:2],
)
print(f"Val DS 2: {ds.answers}")

Train DS: ['rural', 'between 101 and 1000', 'between 11 and 100']
Val DS 1: ['rural', 'between 101 and 1000']
Val DS 2: ['rural', 'between 101 and 1000']

RSVQALRDataModule#

This class is a Lightning Data Module, that wraps the RSVQALRDataSet. It automatically generates DataLoader per split with augmentations, shuffling, etc., depending on the split. All images are resized and normalized and images in the train set additionally basic-augmented via noise and flipping/rotation. The train split is also shuffled, however this can be overwritten (see below). To use a DataModule, the setup() function has to be called. This populates the Dataset splits inside the DataModule. Depending on the stage (‘fit’, ‘test’ or None), the setup will prepare only train & validation Dataset, only test Dataset or all three.

from configilm.extra.DataModules import RSVQALR_DataModule

dm = RSVQALR_DataModule.RSVQALRDataModule(
    data_dirs=my_data_path  # path to dataset
)

print("Before:")
print(dm.train_ds)
print(dm.val_ds)
print(dm.test_ds)

Before:
None
None
None

dm.setup(stage="fit")

print("After:")
print(dm.train_ds)
print(dm.val_ds)
print(dm.test_ds)

After:
<configilm.extra.DataSets.RSVQALR_DataSet.RSVQALRDataSet object at 0x7f6ee4cf9b70>
<configilm.extra.DataSets.RSVQALR_DataSet.RSVQALRDataSet object at 0x7f6ee4cbfe20>
None

Afterwards the pytorch DataLoader can be easily accessed. Note, that \(len(DL) = \lceil \frac{len(DS)}{batch\_size} \rceil\), therefore here with the default batch_size of 16: 25/16 -> 2.

train_loader = dm.train_dataloader()
print(len(train_loader))

The DataModule has in addition to the DataLoader settings a parameter each for data_dir, image_size and max_len which are passed through to the DataSet.

DataLoader settings#

The DataLoader have four settable parameters: batch_size, num_workers_dataloader, shuffle and pin_memory with 16, os.cpu_count() / 2, None and None as their default values.

A shuffle of None means, that the train set is shuffled but validation and test are not. Pinned Memory will be set if a CUDA device is found, otherwise it will be of. However, this behaviour can be overwritten with pin_memory. Changing some of these settings will be accompanied by a Message-Hint printed.

dm = RSVQALR_DataModule.RSVQALRDataModule(
    data_dirs=my_data_path,  # path to dataset
    batch_size=4,
    tokenizer=tokenizer
)
dm.setup(stage="fit")

print(len(dm.train_dataloader()))

_ = RSVQALR_DataModule.RSVQALRDataModule(
    data_dirs=my_data_path,  # path to dataset
    shuffle=False
)

_ = RSVQALR_DataModule.RSVQALRDataModule(
    data_dirs=my_data_path,  # path to dataset
    num_workers_dataloader=2
)

_ = RSVQALR_DataModule.RSVQALRDataModule(
    data_dirs=my_data_path,  # path to dataset
    pin_memory=False
)