ersilia.publish.test.services package

Submodules

ersilia.publish.test.services.checks module

class ersilia.publish.test.services.checks.CheckService(logger: Any, model_id: str, dir: str, from_github: bool, from_dockerhub: bool, ios: IOService)

Bases: object

Service for performing various checks on the model.

Parameters:

• logger (logging.Logger) – Logger for logging messages.

• model_id (str) – Identifier of the model.

• dir (str) – Directory where the model repository is located.

• from_github (bool) – Flag indicating whether to fetch the repository from GitHub.

• from_dockerhub (bool) – Flag indicating whether to fetch the repository from DockerHub.

• ios (IOService) – Instance of IOService for handling input/output operations.

Examples

check_service = CheckService(
    logger=logger,
    model_id="model_id",
    dir="/path/to/dir",
    from_github=True,
    from_dockerhub=False,
    ios=ios,
)
check_service.check_files()

check_consistent_output(run_example, run_model)

Check if the model produces consistent output.

Parameters:

• run_example (callable) – Function to generate example input.

• run_model (callable) – Function to run the model.

check_dim()

check_files()

Check the existence of required files for the model.

check_information()

Perform various checks on the model information.

Parameters:

output (file-like object) – The output file to write to.

check_model_output_content(run_example, run_model)

check_robustness(run_example, run_model)

Robustness test for the Simple Run Model section.

For non-variable (Fixed) models:

• Generate 3 example inputs using the example command.

• Run the model and check that the result is not all null.

For variable models:

• Generate 3 example inputs, run the model, check if all null.

• If all null, generate 5, run again, check if all null.

• If all null, generate 10, run again, check if all null.

• If still all null after all attempts, mark as FAILED.

Parameters:

• run_example (callable) – Function to generate example inputs. Signature: run_example(n_samples, file_name, simple, mode) -> examples

• run_model (callable) – Function to run the model. Signature: run_model(inputs, output, batch) -> result

Returns:

List of (check_name, detail, status) tuples for table rendering.

Return type:

list of tuple

check_simple_model_async_output(serve_model)

check_simple_model_output(run_model)

compare_csv_columns(column_csv, csv_file)

find_missing_first_output_col(path)

get_inputs(types)

trim_string(text, max_length=100)

validate_file_content(file_path, input_type)

Validate output file content and check input data match.

ersilia.publish.test.services.constants module

class ersilia.publish.test.services.constants.Checks(value)

Bases: Enum

Enum for different check types.

COLUMN_CHECK_FAILURE = 'Columns not coincide with run_columns'

COLUMN_CHECK_SUCCESS = 'Columns coincides with run_columns'

COLUMN_MISMATCH = 'Column mismatch check'

COLUMN_NAME_VALIDITY = 'Columns'

CONSISTENCY = 'Model Output Was Consistent'

CSV_CONTENT_CHECK = 'CSV File Content Check'

DEPENDENCY_PINNED = 'Checking package versions and file structure'

DIR_SIZE = 'Directory Size Mb'

EMPTY_COLUMNS = 'Empty Column Found'

ENV_SIZE = 'Environment Size Mb'

FETCH_FAILS = 'Fetch Status'

IMAGE_SIZE = 'Image Size Mb'

INCONSISTENCY = 'Inconsistent Output Detected'

MODEL_CONSISTENCY = 'Check Consistency of Model Output'

PREDEFINED_EXAMPLE = 'Check Predefined Example Input'

ROBUSTNESS_TEST = 'Robustness Test'

RUN_BASH = 'RMSE-MEAN'

SIMPLE_MODEL_RUN = 'Simple Model Run'

SIMPLE_MODEL_RUN_ASYNC = 'Async Simple Model Run'

SIMPLE_MODEL_RUN_COLUMNS = 'Simple Model Run Columns'

SIZE_CACL_FAILED = 'Size Calculation Failed'

SIZE_CACL_SUCCESS = 'Size Successfully Calculated'

TOTAL_DIR_SIZE = 'Total directory size'

class ersilia.publish.test.services.constants.Options(value)

Bases: Enum

Enum for different options.

ASYNC_OUTPUT_CSV = '/root/eos/temp/files/async.csv'

BASE = 'base'

DEEP_INPUT = '/root/eos/temp/files/deep_input.csv'

DEEP_OUTPUT = '/root/eos/temp/files/deep_output.csv'

EXAMPLE_CSV = '/root/eos/temp/files/example.csv'

INPUT_CSV = '/root/eos/temp/files/input.csv'

INPUT_TYPES = ['str', 'list', 'csv']

INPUT_TYPES_TEST = ['csv']

NUM_SAMPLES = 5

OUTPUT1_CSV = '/root/eos/temp/files/output1.csv'

OUTPUT2_CSV = '/root/eos/temp/files/output2.csv'

OUTPUT_CSV = '/root/eos/temp/files/result.csv'

OUTPUT_FILES = ['/root/eos/temp/files/file.csv', '/root/eos/temp/files/file.h5', '/root/eos/temp/files/file.json']

OUTPUT_FILES_TEST = ['/root/eos/temp/files/file.csv']

ROBUSTNESS_INPUT_CSV = '/root/eos/temp/files/robustness_input.csv'

ROBUSTNESS_OUTPUT_CSV = '/root/eos/temp/files/robustness_output.csv'

class ersilia.publish.test.services.constants.STATUS_CONFIGS(value)

Bases: Enum

Enum for status configurations.

FAILED = ('FAILED', 'red', '✘')

NA = ('N/A', 'dim', '~')

NOT_PRESENT = ('NOT PRESENT', 'yellow', '✘')

PASSED = ('PASSED', 'green', '✔')

SKIPPED = ('SKIPPED', 'dim', '~')

SUCCESS = ('SUCCESS', 'green', '★')

WARNING = ('WARNING', 'yellow', '⚠')

class ersilia.publish.test.services.constants.TableConfig(title: str, headers: List[str])

Bases: object

Configuration for a table.

headers: List[str]

title: str

class ersilia.publish.test.services.constants.TableType(value)

Bases: Enum

Enum for different table types.

ASNC_MODEL_RUN_CHECK = 'Async Model Run Check'

COMPUTATIONAL_PERFORMANCE = 'Computational Performance Check'

CONSISTENCY_BASH = 'Consistency Summary Between Ersilia and Bash Execution Outputs'

DEPENDECY_COLUMN_CHECK = 'Dependency and Column Value Checks'

FETCH_STATUS_SURFACE = 'Model Fetching Check'

FINAL_RUN_SUMMARY = 'Test Run Summary'

INSPECT_SUMMARY = 'Inspect Summary'

MODEL_DIM_CHECK = 'Model Dimension Check'

MODEL_DIRECTORY_SIZES = 'Directory Size Check'

MODEL_FILE_CHECKS = 'Model File Checks'

MODEL_INFORMATION_CHECKS = 'Model Metadata Checks'

MODEL_OUTPUT = 'Input Output Check'

MODEL_RUN_CHECK = 'Model Run Check'

MODEL_SIZES = 'Model Size Check'

ROBUSTNESS_CHECK = 'Robustness Check'

RUNNER_CHECKUP_STATUS = 'Runner Checkup Status'

SHALLOW_CHECK_SUMMARY = 'Model Output Consistency Check'

ersilia.publish.test.services.inspect module

class ersilia.publish.test.services.inspect.CheckStrategy(check_function, success_key, details_key)

Bases: object

Execuetd a strategy for checking inspect commands.

Parameters:

• check_function (callable) – The function to check.

• success_key (str) – The key for success.

• details_key (str) – The key for details.

execute()

Execute the check strategy.

Returns:

The results of the check.

Return type:

dict

class ersilia.publish.test.services.inspect.InspectService(dir: str, model: str, remote: bool = False, config_json: str | None = None, credentials_json: str | None = None)

Bases: ErsiliaBase

Service for inspecting models and their configurations.

Parameters:

• dir (str, optional) – Directory where the model is located.

• model (str, optional) – Model identifier.

• remote (bool, optional) – Flag indicating whether the model is remote.

• config_json (str, optional) – Path to the configuration JSON file.

• credentials_json (str, optional) – Path to the credentials JSON file.

Examples

inspector = InspectService(
    dir="/path/to/model", model="model_id"
)
results = inspector.run()

run(check_keys: list | None = None) → dict

Run the inspection checks on the specified model.

Parameters:

check_keys (list, optional) – A list of check keys to execute. If None, all checks will be executed.

Returns:

A dictionary containing the results of the inspection checks.

Return type:

dict

Raises:

• ValueError – If the model is not specified.

• KeyError – If any of the specified keys do not exist.

class ersilia.publish.test.services.inspect.ModelInspector(model: str, dir: str, config_json=None)

Bases: object

Class for inspecting model repositories.

Parameters:

• model (str) – The ID of the model to be inspected.

• dir (str) – The directory where the model repository is located.

• config_json (str, optional) – Path to the configuration JSON file.

Examples

inspector = ModelInspector(
    model="model_id", dir="path/to/repo"
)
result = inspector.check_repo_exists()
result = inspector.check_complete_metadata()

ERSILIAPACK_FILES = ['model/framework/run.sh', 'README.md', 'LICENSE', 'install.yml', 'metadata.yml', 'model/framework/examples/run_input.csv', 'model/framework/examples/run_output.csv', 'model/framework/columns/run_columns.csv', 'model/framework/run.sh', 'README.md', 'LICENSE']

REQUIRED_FIELDS = ['Publication', 'Source Code', 'S3', 'DockerHub']

check_complete_folder_structure()

Check if the folder structure of the repository is complete.

Returns:

A namedtuple containing the success status and details of the check.

Return type:

Result

check_complete_metadata()

Check if the metadata file is complete.

Returns:

A namedtuple containing the success status and details of the check.

Return type:

Result

check_computational_performance()

Check the computational performance of the model.

Returns:

A namedtuple containing the success status and details of the check.

Return type:

Result

check_dependencies_are_valid()

Check if the dependencies in the Dockerfile or install.yml are valid.

For , the Dockerfile is validated. For PACK_METHOD_FASTAPI, either file is acceptable. If one exists, it is validated; if both exist, both are validated. If neither is present, an error is returned.

Returns:

A namedtuple containing the success status and details of the check.

Return type:

Result

check_no_extra_files()

Check if there are no extra files in the repository.

Returns:

A namedtuple containing the success status and details of the check.

Return type:

Result

check_repo_exists()

Check if the model repository exists.

Returns:

A namedtuple containing the success status and details of the check.

Return type:

Result

validate_repo_structure()

Validate the repository structure.

Return type:

List of missing items.

class ersilia.publish.test.services.inspect.Result(success, details)

Bases: tuple

details

Alias for field number 1

success

Alias for field number 0

ersilia.publish.test.services.io module

class ersilia.publish.test.services.io.IOService(logger, model_id: str, dir: str, from_dir: str | None = None)

Bases: object

Service for handling input/output operations related to model testing.

Parameters:

• logger (logging.Logger) – Logger for logging messages.

• model_id (str) – Identifier of the model.

• dir (str) – Directory where the model repository is located.

Examples

ios = IOService(
    logger=logger,
    model_id="model_id",
    dir="/path/to/dir",
)
ios.read_metadata()

calculate_directory_size(path: str) → int

Calculate the size of a directory.

Parameters:

path (str) – The path to the directory.

Returns:

The size of the directory.

Return type:

int

calculate_image_size(tag='latest')

Calculate the size of a Docker image.

Parameters:

tag (str, optional) – The tag of the Docker image (default is ‘latest’).

Returns:

The size of the Docker image.

Return type:

str

collect_and_save_json(results, output_file, from_dockerhub, deep)

Helper function to collect JSON results and save them to a file.

get_conda_env_size()

Get the size of the Conda environment for the model.

Returns:

The size of the Conda environment in megabytes.

Return type:

int

Raises:

Exception – If there is an error calculating the size.

get_directories_sizes(**kwargs)

get_env_sizes(**kwargs)

get_file_requirements()

Get the list of required files for the model.

Returns:

List of required files.

Return type:

List[str]

Raises:

ValueError – If the model type is unsupported.

get_model_dim()

Get the model dimension from the metadata and the run column file

Returns:

A dimensions from the metadata and the run column file

Return type:

tuple

static get_model_type(model_id: str, repo_path: str) → str

Get the type of the model based on the repository contents.

Parameters:

• model_id (str) – Identifier of the model.

• repo_path (str) – Path to the model repository.

Returns:

The type of the model (e.g., PACK_METHOD_FASTAPI).

Return type:

str

get_output_consistency()

static read_csv_values(file_path)

Reads a CSV file and returns its values excluding the header row.

Parameters:

file_path (str) – The path to the CSV file.

Returns:

A list of rows (each row is a list of values) from the CSV file without the header.

Return type:

list

submit_smiles_and_get_results(input, cmd_out, poll_interval=2)

class ersilia.publish.test.services.io.PackageInstaller(dir_path, model_id)

Bases: object

install_packages_from_dir()

ersilia.publish.test.services.parser module

class ersilia.publish.test.services.parser.DockerfileInstallParser(file_dir, conda_env_name=None)

Bases: InstallParser

class ersilia.publish.test.services.parser.InstallParser(file_name, conda_env_name=None)

Bases: object

check_file_exists()

get_python_exe()

write_bash_script(file_name=None)

class ersilia.publish.test.services.parser.YAMLInstallParser(file_dir, conda_env_name=None)

Bases: InstallParser

ersilia.publish.test.services.parser.build_python_executable_script(env)

ersilia.publish.test.services.parser.conda_python_executable(env=None)

ersilia.publish.test.services.parser.eval_conda_prefix() → str

ersilia.publish.test.services.parser.get_conda_source(env)

ersilia.publish.test.services.parser.get_native()

ersilia.publish.test.services.parser.get_python_executable_from_env(env)

ersilia.publish.test.services.parser.run_bash(script)

ersilia.publish.test.services.runner module

class ersilia.publish.test.services.runner.RunnerService(model_id: str, logger, ios_service: IOService, checkup_service: CheckService, setup_service: SetupService, dir: str, from_dir: str, model_path: Callable, from_github: bool, from_s3: bool, from_dockerhub: bool, version: str, shallow: bool, deep: bool, report_path: str, inspector: InspectService, surface: bool, inspect: bool)

Bases: object

Service for running model tests and checks.

Parameters:

• model_id (str) – Identifier of the model.

• logger (logging.Logger) – Logger for logging messages.

• ios_service (IOService) – Instance of IOService for handling input/output operations.

• checkup_service (CheckService) – Instance of CheckService for performing various checks on the model.

• setup_service (SetupService) – Instance of SetupService for setting up the environment and fetching the model repository.

• dir (str) – Directory where the model repository is located.

• from_dir (str) – Directory where the model repository is located.

• model_path (Callable) – Callable to get the model path.

• from_github (bool) – Flag indicating whether to fetch the repository from GitHub.

• from_s3 (bool) – Flag indicating whether to fetch the repository from S3.

• from_dockerhub (bool) – Flag indicating whether to fetch the repository from DockerHub.

• version (str) – Version of the model.

• shallow (bool) – Flag indicating whether to perform shallow checks.

• deep (bool) – Flag indicating whether to perform deep checks.

• report_path (bool) – Flag to specify the path for output as json.

• inspector (InspectService) – Instance of InspectService for inspecting models and their configurations.

• surface (bool) – Flag indicating whether to perform surface checks.

• inspect (bool) – Flag indicating whether to perform inspect checks.

delete()

Delete model if existed

fetch()

Fetch the model repository from the specified directory.

run()

Run the model tests and checks.

run_bash()

Run the model using a bash script and compare the outputs for consistency.

Raises:

RuntimeError – If there is an error during the subprocess execution or output comparison.

run_example(n_samples: int, file_name: str | None = None, simple: bool = True, mode: str = 'random')

Generate example input samples for the model.

Parameters:

• n_samples (int) – Number of samples to generate.

• file_name (str, optional) – Name of the file to save the samples.

• simple (bool, optional) – Flag indicating whether to generate simple samples.

• mode (str, optional) – Mode for generating samples. Can be “predefined”, “deterministic”, or “random” (default = “random”).

Returns:

List of generated input samples.

Return type:

list

run_model(inputs: str, output: str, batch: int)

Run the model with the given input and output parameters.

Parameters:

• input (str) – A path to input file

• output (str) – Path to the output file.

• batch (int) – Batch size for running the model.

Returns:

The output of the command.

Return type:

str

serve_model()

Serve the model and return the command output.

Returns:

The output of the command.

Return type:

str

ersilia.publish.test.services.setup module

class ersilia.publish.test.services.setup.SetupService(model_id: str, dir: str, from_github: bool, from_s3: bool, logger: Any)

Bases: object

Service for setting up the environment and fetching the model repository.

Parameters:

• model_id (str) – Identifier of the model.

• dir (str) – Directory where the model repository will be cloned.

• from_github (bool) – Flag indicating whether to fetch the repository from GitHub.

• from_s3 (bool) – Flag indicating whether to fetch the repository from S3.

• logger (Any) – Logger for logging messages.

BASE_URL = 'https://github.com/ersilia-os/'

static get_conda_env_location(model_id: str, logger) → str

Get the location of the Conda environment for the model.

Parameters:

• model_id (str) – Identifier of the model.

• logger (logging.Logger) – Logger for logging messages.

Returns:

The location of the Conda environment.

Return type:

str

Raises:

subprocess.CalledProcessError – If the command to list Conda environments returns a non-zero exit code.

get_model()

static run_command(command, logger, capture_output: bool = False, shell: bool = True, env=None, check: bool = False)

Run a shell command.

Parameters:

• command (str | list | tuple) – The command to run.

• logger (logging.Logger) – Logger for logging messages.

• capture_output (bool, optional) – Flag indicating whether to capture the command output.

• shell (bool, optional) – Flag indicating whether to run the command in the shell.

• env (dict | None, optional) – Environment variables to use (defaults to os.environ).

• check (bool, optional) – If True, raise CalledProcessError on non-zero exit codes (capture_output=True only).

Returns:

If capture_output=True returns stdout as string. Otherwise returns process.stdout (a stream).

Return type:

str | io.TextIOBase

Module contents