CalistaEngine

class calista.table.CalistaEngine(engine: str, config: Dict[str, Any] | None = None)

Bases: object

For now, you can execute data quality checks using the following engines or platforms: spark, pandas, polars, snowflake, bigquery.

load(path: str | None = None, file_format: str | None = None, data: Dict[str, List] | None = None, table: str | None = None, schema: str | None = None, database: str | None = None, dataframe: Any | None = None, options: Dict[str, Any] | None = None) → CalistaTable

Load data from a dataset into a CalistaTable.

Parameters:

• path ((str, optional)) – The path if you’re loading a file.

• file_format ((str, optional)) – The format of the file (e.g., ‘csv’, ‘parquet’).

• data ((dict)) – The dictionary containing the data of the table.

• table ((str, optional)) – The name of the table if you’re not loading a file.

• schema ((str, optional)) – The schema containing the table.

• database ((str, optional)) – The database containing the table.

• dataframe ((Any, optional)) – An existing dataframe.

• options ((Dict[str, Any], optional)) – Additional configuration file options.

Returns:

CalistaTable: The loaded table.

Raises:

Any exceptions raised by the engine’s read_dataset method.

load_from_database(table: Any, schema: str | None = None, database: str | None = None) → CalistaTable

Load data from a table into a CalistaTable.

Parameters:

• table ((str)) – The name of the table.

• schema ((str, optional)) – The schema containing the table

• database ((str, optional)) – The database containing the table.

Returns:

CalistaTable: The loaded table.

Raises:

Any exceptions raised by the engine’s read_dataset method.

load_from_dataframe(dataframe: Any) → CalistaTable

Load data from a dataframe into a CalistaTable.

Parameters:

dataframe ((Any)) – An existing dataframe.

Returns:

CalistaTable: The loaded table.

Raises:

Any exceptions raised by the engine’s read_dataset method.

load_from_dict(data: Dict[str, List]) → CalistaTable

Load data from a dictionary into a CalistaTable.

Parameters:

data ((dict)) – The dictionary containing the data of the table.

Returns:

CalistaTable: The loaded table.

Raises:

Any exceptions raised by the engine’s read_dataset method.

Example

>>> calista_table = CalistaEngine(engine = "spark").load_from_dict({"ID": [1, 2, 3, 4]})
>>> calista_table.show()
+---+
| ID|
+---+
|  1|
|  2|
|  3|
|  4|
+---+

load_from_path(path: str, file_format: str, options: Dict[str, Any] | None = None) → CalistaTable

Load data from a path into a CalistaTable.

Parameters:

• path ((str)) – The path of the file containing your table.

• file_format ((str, optional)) – The format of the file (e.g., ‘csv’, ‘parquet’).

Returns:

CalistaTable: The loaded table.

Raises:

Any exceptions raised by the engine’s read_dataset method.

Example

>>> csv_options = {
>>> "delimiter": ",",
>>> "header": "True"
>>> }
>>> calista_table = CalistaEngine(engine = "spark").load_from_path(path='my_csv.csv',file_format="csv",options=csv_options)