Deployment¶

⚙️ First-Time Setup: Intelligence Layer + DataClay¶

🔧 Step 1: Start dataClay Backend Services¶

docker compose restart dataclay-backend
docker-compose up       # Start all services
docker-compose down     # Stop all services

🛠️ Step 2: Build and Containerize Your Model¶

bentoml build -f ./bentofile.yaml
bentoml containerize analytics:ID  # Replace ID with your model tag

¶

🧠 Running the Model with Docker¶

🖥️ Runtime Configuration via BentoML¶

Use the BENTOML_CONFIG_OPTIONS environment variable to control runtime behavior:

-e BENTOML_CONFIG_OPTIONS='api_server.traffic.timeout=600 runners.resources.cpu=0.5 runners.resources."nvidia.com/gpu"=0'

Explanation: - api_server.traffic.timeout=600 → Extends request timeout to 600 seconds
- runners.resources.cpu=0.5 → Allocates 0.5 CPUs per runner
- runners.resources."nvidia.com/gpu"=0 → Assigns GPU #0 (omit for CPU-only)

🧠 Resource Allocation¶

--cpus 7.5 --memory 14g

Explanation: - --cpus 7.5 → Limits container to 7.5 CPUs
- --memory 14g → Allocates 14 GB of RAM

🌐 Port Mapping¶

-p 3000:3000 -p 5000:5000

Explanation:

• 3000 → BentoML REST API
• 5000 → BentoML gRPC API

🐳 Run Model Container (GPU or CPU)¶

docker run --network host -it --rm \
  -p 3000:3000 -p 5000:5000 \
  --cpus 7.5 --memory 14g \
  -e BENTOML_CONFIG_OPTIONS='api_server.traffic.timeout=600 runners.resources.cpu=0.5 runners.resources."nvidia.com/gpu"=0' \
  analytics:ID serve  # Replace ID with your container tag

✅ For CPU-only, remove runners.resources."nvidia.com/gpu" from the config.

Docker Installation¶

The ICOS Intelligence Layer can be deployed using Docker containers. While the core installation remains consistent with deliverable D4.1, this version introduces new environment configurations to customize API timeout and resource allocation.

Download the Docker image from this link and run the container using the command below:

docker run --network host -it --rm -p 3000:3000 -p 5000:5000 \
-e BENTOML_CONFIG_OPTIONS='api_server.traffic.timeout=600 runners.resources.cpu=0.5 runners.resources."nvidia.com/gpu"=0' \
docker_name:latest serve

This command launches the ICOS Intelligence Layer Coordination API server, accessible via http://0.0.0.0:3000.

Trustworthy AI Module – Deployment¶

Explainable AI¶

• Deployment: Packaged inside the Intelligence Layer Docker container.
• Access: Trigger using the start_mlflow_ui() API endpoint (runs on port 5000).
• URL: http://127.0.0.1:5000
• Dependencies: MLFlow 2.13.2, Python 3.10, BentoML 1.2.16

Prediction Confidence Scores¶

• Tech Stack: Python 3.10, Scikit-learn
• Deployment: Returned as part of model prediction responses.

Model Monitoring (Drift Detection)¶

• API Endpoint: POST /detect_drift
• Description: Activates drift detection using Jensen-Shannon Divergence.
• Input: Boolean flag (true to enable, false to skip).
• Output: Structured report detailing drift metrics and anomalies.

Federated Learning¶

• Tech Stack: Python 3.10, Flower FL, PyTorch
• Model: LSTM for time-series metrics (CPU, memory, power)
• Pipeline: Scaphandre → Prometheus → OpenTelemetry → dataClay
• Export Format: BentoML
• Deployment: Via Docker Compose, supporting multi-service orchestration
• Retraining: Supports periodic or on-demand loops based on data availability

AI Analytics Module – Deployment¶

AIOps with MLFlow¶

• Access: Launch via the start_mlflow_ui() API endpoint (port 5000)
• URL: http://localhost:5000
• Dependencies: MLFlow 2.13.2, Python 3.10, BentoML 1.2.16

System Requirements¶

The following packages and versions are required for successful deployment of the ICOS Intelligence Layer container: