RSVQAxBEN#

This page describes the usage of Dataloader and Datamodule for RSVQAxBEN, a VQA dataset based on the BigEarthNet v1.0 Dataset. It was first published by Lobry et al. [3]. The dataset can be found on zenodo . For the usage in this context, the dataset was extended from its RGB form to include all bands originally included in BigEarthNet v1.0. A small example of the data used is distributed with this package.

This module contains two classes, a standard torch.util.data.Dataset and a pytorch_lightning.LightningDataModule that encapsulates the Dataset for easy use in pytorch_lightning applications. The Dataset uses a BENLMDBReader to read images a LMDB file. Questions and Answers are read using JSON files.

RSVQAxBENDataSet#

In its most basic form, the Dataset only needs the base path of the LMDB file and json files. Note, that from an os point of view, LMDB files are folders. The path should follow the same structure as it is when downloaded from the official page and extracted with images, questions and answer next to each other. The official naming for files is expected.

The full data path structure expected is

datapath = {
    "images_lmdb": "/path/to/BigEarthNetEncoded.lmdb",
    "train_data": "/path/to/train-data",
    "val_data": "/path/to/val-data",
    "test_data": "/path/to/test-data"
}

Note, that the keys have to match exactly while the paths can be selected freely.

from configilm import util
util.MESSAGE_LEVEL = util.MessageLevel.INFO  # use INFO to see all messages

from configilm.extra.DataSets import RSVQAxBEN_DataSet
 
ds = RSVQAxBEN_DataSet.RSVQAxBENDataSet(
    data_dirs=my_data_path  # path to dataset
)

img, question, answer = ds[10]
img = img[:3] # only choose RGB channels
# reorder channels for display
b, g, r = img
img = torch.stack([r, g, b])

Size: torch.Size([3, 120, 120])
Question: are some coniferous forest present in the scene?
Question (start): [101, 2024, 2070, 9530, 23930, 3224, 2556, 1999, 1996, 3496, 1029, 102, 0, 0, 0]
Answer: yes
Answer (start): tensor([0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])

../../_images/c871b0846fbafcd934d0bcb6a8850282ea4e3a9f233f703e297007c86d67b06c.png

Tokenizer and Tokenization#

As we can see, this Dataset uses a tokenizer to generate the Question out of a natural language text. If no tokenizer is provided, a default one will be used, however this may lead to bad performance if not accounted for. The tokenizer can be configured as input parameter.

from configilm.ConfigILM import _get_hf_model

tokenizer, _ = _get_hf_model("prajjwal1/bert-tiny")

ds = RSVQAxBEN_DataSet.RSVQAxBENDataSet(
    data_dirs=my_data_path,  # path to dataset
    tokenizer=tokenizer
)
img, question, answer = ds[0]

Tip

Usually this tokenizer is provided by the model itself as shown in the VQA example during dataset creation.

During tokenization a sequence of tokens (integers) of specific length is generated. The length of this sequence can be set with the parameter seq_length. If the generated tokens are shorter than the sequence length, the sequence will be padded with zeros. If it is longer, the sequence is truncated.

Note

Most tokenizer use an ‘End of Sequence’ token that will always be the last one in the non-padded sequence.

ds = RSVQAxBEN_DataSet.RSVQAxBENDataSet(
    data_dirs=my_data_path,  # path to dataset
    tokenizer=tokenizer,
    seq_length=16
)
_, question1, _ = ds[0]
print(question1)

[101, 2024, 7976, 2752, 2030, 4910, 2752, 2556, 1029, 102, 0, 0, 0, 0, 0, 0]

ds = RSVQAxBEN_DataSet.RSVQAxBENDataSet(
    data_dirs=my_data_path,  # path to dataset
    tokenizer=tokenizer,
    seq_length=8
)
_, question2, _ = ds[0]
print(question2)

[101, 2024, 7976, 2752, 2030, 4910, 2752, 102]

The tokenizer can also be used to reconstruct the input/question from the IDs including the special tokens:

print(f"Question 1: '{tokenizer.decode(question1)}'")
print(f"Question 2: '{tokenizer.decode(question2)}'")

Question 1: '[CLS] are artificial areas or agricultural areas present? [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]'
Question 2: '[CLS] are artificial areas or agricultural areas [SEP]'

or without:

print(f"Question 1: '{tokenizer.decode(question1, skip_special_tokens=True)}'")
print(f"Question 2: '{tokenizer.decode(question2, skip_special_tokens=True)}'")

Question 1: 'are artificial areas or agricultural areas present?'
Question 2: 'are artificial areas or agricultural areas'

Selecting Bands#

Like for the BigEarthNet v1.0 DataSet, this DataSet supports different Band combinations. Currently, the selection is limited to some preconfigured combinations. Which bands are used is defined by the number of channels set in the Dataset. The selection is the same as for the BigEarthNet v1.0 DataSet as we can see when we use a faulty configuration.

try:
    ds = RSVQAxBEN_DataSet.RSVQAxBENDataSet(
        data_dirs=my_data_path,  # path to dataset
        img_size=(-1, 120, 120)
    )
except AssertionError as a:
    print(a)

Splits#

It is possible to load only a specific split ('train', 'val' or 'test') in the dataset. The images loaded are specified using the csv files in the same folder as the LMDB file. By default (None), all three are loaded into the same Dataset.

_ = RSVQAxBEN_DataSet.RSVQAxBENDataSet(
    data_dirs=my_data_path,  # path to dataset
    split="test",
    tokenizer=tokenizer
)

Restricting the number of loaded images#

It is also possible to restrict the number of images indexed. By setting max_img_idx = n only the first n images (in alphabetical order based on their S2-name) will be loaded. A max_img_idx of None, -1 or larger than the number of images in the csv file(s) (in this case 25) equals to load-all-images behaviour.

_ = RSVQAxBEN_DataSet.RSVQAxBENDataSet(
    data_dirs=my_data_path,  # path to dataset
    max_len=10,
    tokenizer=tokenizer
)

_ = RSVQAxBEN_DataSet.RSVQAxBENDataSet(
    data_dirs=my_data_path,  # path to dataset
    max_len=100,
    tokenizer=tokenizer
)

Select Number of Classes or specific Answers#

For some applications, it is relevant to have only a certain number of classes as valid output. For example, the RSVQAxBEN DataSet could contain up to 2^19 > 0.5 million different classes. To prevent this dimension explosion, the number of classes can be limited. For the ‘train’ split, it is then automatically determined which combination of classes results in the highest reduction of the dataset.

train_ds = RSVQAxBEN_DataSet.RSVQAxBENDataSet(
    data_dirs=my_data_path,  # path to dataset
    split="train",
    tokenizer=tokenizer,
    num_classes=3
)

These selected answers can be re-used in other splits or limited if only a subset is required.

Note

The number of classes does not necessarily match the number of answers. If there are fewer answers then classes, the last classes will never be encoded in the one-hot encoded answer vector. If there are more, an IndexError will happen during accessing a non encode-able element.

print(f"Train DS: {train_ds.answers}")

ds = RSVQAxBEN_DataSet.RSVQAxBENDataSet(
    data_dirs=my_data_path,  # path to dataset
    split="val",
    tokenizer=tokenizer,
    selected_answers=train_ds.answers
)
print(f"Val DS 1: {ds.answers}")

ds = RSVQAxBEN_DataSet.RSVQAxBENDataSet(
    data_dirs=my_data_path,  # path to dataset
    split="val",
    tokenizer=tokenizer,
    selected_answers=train_ds.answers[:2],
)
print(f"Val DS 2: {ds.answers}")

Train DS: ['no', 'yes', 'INVALID']
Val DS 1: ['no', 'yes']
Val DS 2: ['no', 'yes']

RSVQAxBENDataModule#

This class is a Lightning Data Module, that wraps the RSVQAxBENDataSet. It automatically generates DataLoader per split with augmentations, shuffling, etc., depending on the split. All images are resized and normalized and images in the train set additionally basic-augmented via noise and flipping/rotation. The train split is also shuffled, however this can be overwritten (see below). To use a DataModule, the setup() function has to be called. This populates the Dataset splits inside the DataModule. Depending on the stage (‘fit’, ‘test’ or None), the setup will prepare only train & validation Dataset, only test Dataset or all three.

from configilm.extra.DataModules import RSVQAxBEN_DataModule

dm = RSVQAxBEN_DataModule.RSVQAxBENDataModule(
    data_dirs=my_data_path  # path to dataset
)

print("Before:")
print(dm.train_ds)
print(dm.val_ds)
print(dm.test_ds)

Before:
None
None
None

dm.setup(stage="fit")

print("After:")
print(dm.train_ds)
print(dm.val_ds)
print(dm.test_ds)

After:
<configilm.extra.DataSets.RSVQAxBEN_DataSet.RSVQAxBENDataSet object at 0x7ff21357e950>
<configilm.extra.DataSets.RSVQAxBEN_DataSet.RSVQAxBENDataSet object at 0x7ff213f92da0>
None

Afterwards the pytorch DataLoader can be easily accessed. Note, that \(len(DL) = \lceil \frac{len(DS)}{batch\_size} \rceil\), therefore here with the default batch_size of 16: 25/16 -> 2.

train_loader = dm.train_dataloader()
print(len(train_loader))

The DataModule has in addition to the DataLoader settings a parameter each for data_dir, image_size and max_img_idx which are passed through to the DataSet.

DataLoader settings#

The DataLoader have four settable parameters: batch_size, num_workers_dataloader, shuffle and pin_memory with 16, os.cpu_count() / 2, None and None as their default values.

A shuffle of None means, that the train set is shuffled but validation and test are not. Pinned Memory will be set if a CUDA device is found, otherwise it will be of. However, this behaviour can be overwritten with pin_memory. Changing some of these settings will be accompanied by a Message-Hint printed.

dm = RSVQAxBEN_DataModule.RSVQAxBENDataModule(
    data_dirs=my_data_path,  # path to dataset
    batch_size=4,
    tokenizer=tokenizer
)
dm.setup(stage="fit")

print(len(dm.train_dataloader()))

_ = RSVQAxBEN_DataModule.RSVQAxBENDataModule(
    data_dirs=my_data_path,  # path to dataset
    shuffle=False
)

_ = RSVQAxBEN_DataModule.RSVQAxBENDataModule(
    data_dirs=my_data_path,  # path to dataset
    num_workers_dataloader=2
)

_ = RSVQAxBEN_DataModule.RSVQAxBENDataModule(
    data_dirs=my_data_path,  # path to dataset
    pin_memory=False
)