Dataset¶

The Dataset class wraps pandas DataFrames for use in Virtualitics apps.

Dataset Class¶

Dataset ¶

Dataset(dataset: DataFrame, label: str, metadata: Optional[dict] = None, name: Optional[str] = None, encoding: Union[str, DataEncoding] = 'ordinal', one_hot_dict: Optional[Dict[str, List[str]]] = None, cat_to_vals: Optional[Dict[str, List[object_t]]] = None, categorical_cols: Optional[List[str]] = None, predict_cols: Optional[List[str]] = None, description: Optional[str] = None, version: Optional[int] = None, **kwargs)

The dataset asset allows for easy conversion between data formats when it is provided with additional inputs to convert between formats.

Parameters:

dataset (DataFrame) –

A dataset containing numerical and categorical columns. Should be given as a pandas DataFrame.
label (str) –

Label for Asset. See Asset documentation for more details.
metadata (Optional[dict], default: None ) –

Asset metadata. See asset documentation for more details.
name (Optional[str], default: None ) –

Name for Asset. See Asset documentation for more details.
encoding (Union[str, DataEncoding], default: 'ordinal' ) –

DataEncoding enum to specify the data type of the given dataset. Possible values are ordinal,one_hot, or verbose. These types refer to the format of categorical features. ordinal means that categories are contained only in a single column and encoded with integers. verbose is the same format, but encoded with strings instead of integers. one_hot means categorical features are split up into columns for each possible value using a one-hot-encoding methodology.
one_hot_dict (Optional[Dict[str, List[str]]], default: None ) –

Allows conversion to and from 'one_hot' encoding. This is a dictionary mapping from names of categorical features to a list of strings of the columns in the dataset which correspond to the given feature. If not provided and the dataset is given in a one hot encoding, attempts to create one_hot_dict assuming that the columns were created using pd.get_dummies.
cat_to_vals (Optional[Dict[str, List[object_t]]], default: None ) –

This is a dictionary mapping from names of categorical features to a list of strings representing their possible values. Even if not provided, one is inferred from the given dataset.
categorical_cols (Optional[List[str]], default: None ) –

A list of strings representing the feature names of the category features. For an ordinal or verbose encoded dataset, it would just be the name of the column of the categorical feature. For a one_hot encoded dataset, it would be the corresponding name of the feature.
predict_cols (Optional[List[str]], default: None ) –

Names of columns in the dataset that will be used by a model. This allows filtering of the dataframe when passing to a model even if the dataset contains extraneous columns. These columns are expected to match the provided encoding of the dataset.
description (Optional[str], default: None ) –

Description of :class:~virtualitics_sdk.assets.asset.Asset, see its documentation for more details.
version (Optional[int], default: None ) –

Version of :class:~virtualitics_sdk.assets.asset.Asset, see its documentation for more details. EXAMPLE: .. code-block:: python # Imports from virtualitics_sdk import Dataset . . . # Example usage df = store_interface.get_element_value(data_upload_step.name, "Upload data here!") vaip_dataset = Dataset(df, "ExampleDatasetLabel", name="ExampleData")

filter_data ¶

filter_data(X: DataFrame, encoding: Optional[Union[str, DataEncoding]] = None)

Filters columns of the provided dataframe so that they contain only columns used for model prediction. This filtering is only possible when this Dataset object was initialized with the predict_cols parameter.

Parameters:

X (DataFrame) –

The dataframe which will be filtered. This dataframe should contain every column specified in the intialization of the predict_cols parameter.
encoding (Optional[Union[str, DataEncoding]], default: None ) –

The encoding of the provided dataframe. If None, it assumes the dataframe is in the same encoding as the original provided dataframe. Can also be provided as a string version of the encoding. Defaults to None.

Returns:

–

The filtered dataset.

convert_dtypes ¶

convert_dtypes(X: DataFrame)

Converts the dtypes of the columns in X to match the dtypes of this asset's dataset object.

Parameters:

X (DataFrame) –

The dataframe whose dtypes will be converted.

Returns:

–

The same dataframe with converted dtypes.

convert_encoding ¶

convert_encoding(X: DataFrame, from_: Optional[Union[str, DataEncoding]] = None, to_: Optional[Union[str, DataEncoding]] = None, filter: bool = False) -> pd.DataFrame

Converts the dataframe from and to the specified encodings. The dataframe should be in the encoding specified in from_. The dataframe can also be concurrently filtered to only contain prediction columns.

Parameters:

X (DataFrame) –

The dataframe to be converted. Should be in the encoding specified in from_.
from_ (Optional[Union[str, DataEncoding]], default: None ) –

The encoding of the provided dataframe. If None, it assumes the dataframe is in the same encoding as the original provided dataframe. Can also be provided as a string version of the encoding. Defaults to None.
to_ (Optional[Union[str, DataEncoding]], default: None ) –

The encoding to convert the dataframe to. If None, it assumes the dataframe is in the same encoding as the original provided dataframe. Can also be provided as a string version of the encoding. Defaults to None.
filter (bool, default: False ) –

Whether to filter the provided data to only contain prediction columns. Defaults to False.

Returns:

DataFrame –

The converted dataframe.

Raises:

ValueError –

When either the from_ or to_ encodings are not supported for conversion.

check_valid_encoding ¶

check_valid_encoding(encoding: Optional[Union[str, DataEncoding]] = None) -> DataEncoding

Converts provided encoding to a :class:~virtualitics_sdk.assets.dataset.DataEncoding enum. If no encoding is provided, defaults to the original encoding of the provided dataframe in initialization.

Parameters:

encoding (Optional[Union[str, DataEncoding]], default: None ) –

The encoding to be validated If None, the functions returns the default encoding provided in initialization. Can also be provided string versions of the encodings. Valid strings are ordinal, one_hot, and verbose. Defaults to None.

Returns:

DataEncoding –

The corresponding :class:~virtualitics_sdk.assets.dataset.DataEncoding enum.

Raises:

ValueError –

If the provided string does not match a valid DataEncoding.

get_as_encoding ¶

get_as_encoding(encoding: Optional[Union[str, DataEncoding]] = None, filter: bool = False) -> pd.DataFrame

Returns this asset's dataset as the specified encoding.

Parameters:

encoding (Optional[Union[str, DataEncoding]], default: None ) –

The encoding to convert the dataframe to. If None, it assumes the dataframe is in the same encoding as the original provided dataframe. Can also be provided as a string version of the encoding. Defaults to None.
filter (bool, default: False ) –

Whether to filter the provided data to only contain prediction columns. Defaults to False.

Returns:

DataFrame –

The dataset in the specified encoding, with additional filtering if specified.

get_categorical_names ¶

get_categorical_names(predict_cols: bool = False) -> List[str]

Returns the names of the categorical columns of this dataset. Can also optionally return only categorical columns which are also prediction columns.

Parameters:

predict_cols (bool, default: False ) –

Whether to reduce the set of categorical columns returned to just prediction columns. Defaults to False.

Returns:

List[str] –

The list of categorical columns.

Basic Usage¶

from virtualitics_sdk import Dataset
import pandas as pd

# Create a DataFrame
df = pd.DataFrame({
    'product': ['A', 'B', 'C'],
    'sales': [100, 200, 150],
    'region': ['North', 'South', 'East']
})

# Wrap in Dataset
dataset = Dataset(
    name="Product Sales",
    data=df
)

# Store for next step
self._outLink.sales_dataset = dataset

Retrieving Datasets¶

def run(self, flow_metadata):
    # Get dataset from previous step
    dataset = self._inLink.sales_dataset

    # Access the underlying DataFrame
    df = dataset.data

    # Work with it as normal pandas DataFrame
    total_sales = df['sales'].sum()
    top_product = df.loc[df['sales'].idxmax(), 'product']

Loading from Files¶

# From CSV
df = pd.read_csv('data.csv')
dataset = Dataset(name="CSV Data", data=df)

# From Excel
df = pd.read_excel('data.xlsx')
dataset = Dataset(name="Excel Data", data=df)

# From database query
df = pd.read_sql(query, connection)
dataset = Dataset(name="Query Results", data=df)

Dataset Operations¶

def run(self, flow_metadata):
    dataset = self._inLink.raw_data
    df = dataset.data

    # Filter
    filtered = df[df['sales'] > 100]

    # Transform
    df['profit'] = df['sales'] * 0.2

    # Aggregate
    summary = df.groupby('region')['sales'].sum()

    # Create new dataset with results
    processed_dataset = Dataset(
        name="Processed Data",
        data=df
    )

    self._outLink.processed = processed_dataset

Metadata¶

Store additional metadata with datasets:

dataset = Dataset(
    name="Sales Data",
    data=df,
    metadata={
        "source": "database",
        "timestamp": datetime.now().isoformat(),
        "row_count": len(df),
        "columns": list(df.columns)
    }
)

# Access metadata later
metadata = dataset.metadata
print(f"Loaded {metadata['row_count']} rows from {metadata['source']}")

Best Practices¶

Descriptive Names: Use clear, descriptive dataset names
Data Types: Ensure correct pandas dtypes before wrapping
Size Limits: Very large datasets (>100MB) may impact performance
Immutability: Treat datasets as immutable, create new ones for modifications
Validation: Validate data before creating datasets

Dataset¶

Dataset Class¶

Dataset ¶

filter_data ¶

convert_dtypes ¶

convert_encoding ¶

check_valid_encoding ¶

get_as_encoding ¶

get_categorical_names ¶

Basic Usage¶

Retrieving Datasets¶

Loading from Files¶

Dataset Operations¶

Metadata¶

Best Practices¶

See Also¶