lightrft.datasets.prompts_dataset_vl

class lightrft.datasets.prompts_dataset_vl.PromptDatasetVL(*args: Any, **kwargs: Any)

Bases: Dataset

A PyTorch Dataset for Vision-Language (VL) prompting tasks.

This class wraps a raw dataset (e.g., a HuggingFace Dataset, list of dicts) and preprocesses each item on-the-fly using the preprocess_data function. It prepares the data in a (prompt, images, reference, label) format suitable for training or evaluation.

Parameters:

• dataset (Any) – The underlying raw dataset (can be a HuggingFace Dataset, list, or pandas-like object)

• tokenizer (Any) – The tokenizer for text processing

• processor (Any) – The processor, which may include the tokenizer and image processor

• max_length (int) – The maximum sequence length for the tokenizer

• strategy (Any) – A configuration object containing keys and flags for data extraction

• input_template (str | None) – Template for formatting input text

collate_fn(batch: List[Tuple]) → Tuple[List, List, List, List]

Collates a batch of preprocessed data items.

Parameters:

batch (List[Tuple]) – A list of tuples, where each tuple is the output of __getitem__

Returns:

A tuple of lists, containing (prompts, images, references, labels)

Return type:

Tuple[List, List, List, List]

lightrft.datasets.prompts_dataset_vl.preprocess_data(data: Dict[str, Any], input_template: str | None = None, prompt_key: str | None = None, images_key: str = 'images', reference_key: str | None = None, label_key: str | None = None, apply_chat_template=None, processor=None, system_prompt: str | None = None) → Tuple[Any, Any, Any, Any]

Extracts and formats prompt, images, reference, and label from a data record.

This function serves as the core pre-processing logic for preparing a single data point for a vision-language model.

Parameters:

• data (Dict[str, Any]) – A single data record as a dictionary

• input_template (str | None) – A template to format the user’s prompt

• prompt_key (str | None) – The key to access the prompt/conversation

• images_key (str) – The key to access images

• reference_key (str | None) – The primary key for the reference/answer

• label_key (str | None) – The key for the label

• apply_chat_template (Callable, optional) – The tokenizer’s apply_chat_template method

• processor (Any, optional) – The model’s processor (used for chat template)

• system_prompt (str | None) – An optional system prompt to prepend

Returns:

A tuple containing the processed (prompt, images, reference, label)

Return type:

Tuple[Any, Any, Any, Any]