Introduction
Epico Client - High-level Python client for the DMS platform
Epico Client
Epico is a high-level Python client library for the DMS platform. It provides a clean, intuitive API for interacting with the DMS Orchestrator and managing data workflows.
Overview
Epico provides a Pythonic interface to the DMS platform, hiding the complexity of the underlying API calls behind simple Python classes and methods. It enables users to manage projects, collections, data sources, and workflows through a familiar Python interface.
Features
Project Management
- Project Creation: Create and manage research projects
- Project Organization: Hierarchical project structures
- Metadata Management: Store and retrieve project metadata
- User Association: Link projects to users and teams
Collection Management
- Collection Creation: Define collections with schemas
- Schema Validation: Enforce data organization rules
- Data Source Registration: Link collections to S3 data sources
- Hook Integration: Custom processing hooks for data transformation
Data Workflows
- Loading Plan Generation: Create execution plans for data processing
- Workflow Execution: Apply loading plans to process data
- Status Tracking: Monitor workflow execution progress
- Error Handling: Comprehensive error management
Data Access
- Presigned URL Generation: Secure access to S3 data
- File Operations: Upload and download files
- Batch Operations: Process multiple files efficiently
- Progress Tracking: Monitor data transfer progress
Installation
From PyPI
bash
pip install miura-epico
From Source
bash
git clone https://gitlab.com/wabisabi1/epico.git
cd epico
poetry install
Quick Start
Authentication
python
import miura
# Get authentication token
token = miura.auth.get_token(username="user@example.com", password="password")
Project Management
python
# List all projects
projects = miura.nexus.list_projects(token)
# Create a new project
project = miura.nexus.create_project(
name="My Research Project",
description="Project for CFD simulations",
token=token
)
Collection Management
python
# Define collection schema
collection_schema = {
"name": "Simulation Schema",
"path": "/",
"description": "Schema for simulation results",
"isRequired": True,
"occurrence": 1,
"children": [
{
"path": "/simulation_\\d{4}",
"description": "Folder for each simulation run",
"isRequired": True,
"occurrence": "any",
"children": [
{
"path": "/simulation_\\d{4}/parameters.dat",
"description": "Simulation parameters file",
"isRequired": True,
"occurrence": 1
},
{
"path": "/simulation_\\d{4}/results.dat",
"description": "Numerical simulation results",
"isRequired": True,
"occurrence": 1
}
]
}
]
}
# Create collection
collection = project.create_collection(
name="SimulationCollection",
schema=collection_schema
)
Data Source Registration
python
# Register S3 data source
s3_ds = miura.data.S3DataSource("s3://my-bucket/simulation-data")
collection.register_data_source(s3_ds)
Hook Registration
python
# Create a data splitting hook
my_hook = miura.nexus.hooks.DatasetFractionSplitHook(
name="splitting_1",
pattern="simulation_*",
fractions=[0.1, 0.7, 0.2]
)
collection.register_before_plan_hook(my_hook)
Data Processing
python
# Generate loading plan
loading_plan = collection.make_loading_plan()
# Apply the plan to process data
loading_plan.apply()
Data Access
python
# Get presigned URL for file access
presigned_url = collection.get_presigned_url("/simulation_0001/results.dat")
# Download file using presigned URL
import requests
response = requests.get(presigned_url)
with open("results.dat", "wb") as f:
f.write(response.content)
New to the DMS platform? Start with our Getting Started guide to learn the fundamentals before diving into advanced features.