modal

Modal - A Serverless Cloud Platform for Python and On-Demand GPU Computing

Skills Overview

Modal is a serverless cloud platform designed specifically for Python, enabling developers to run code on demand in the cloud—especially well-suited for AI/ML workloads. With the Modal SDK, you can define GPU computing, container images, web endpoints, and scheduled jobs using pure Python code, without writing Dockerfiles or YAML configuration files.

Use Cases

1. Deploying and Serving AI/ML Models

Deploy trained machine learning models as scalable Web APIs. Modal supports a range of GPUs from T4 to H200, making it suitable for large language model inference, computer vision model services, and real-time prediction systems. With the @modal.fastapi_endpoint() decorator, a few lines of code are enough to turn your model into an HTTP endpoint, while also benefiting from automatic scaling and sub-second cold starts.

2. GPU-Accelerated Compute Tasks

Run compute-intensive tasks in the cloud that require GPU resources, including model training, fine-tuning, and large-scale inference. Modal offers on-demand GPU computing and supports both single-GPU (H100, A100, L40S) and multi-GPU distributed training. By using @app.function(gpu="H100:4"), you can request 4 H100 GPUs without having to preconfigure any servers.

3. Batch Processing and Data Pipelines

Build scalable data processing pipelines and ETL jobs. Modal’s .map() method automatically distributes tasks across hundreds of containers for parallel execution. Combined with scheduled jobs, you can create recurring workflows for tasks such as data cleaning, feature engineering, or model retraining. Volumes provide persistent storage for saving datasets and intermediate results.

Core Features

1. Serverless Functions and Classes

Define functions and classes to run in the cloud using decorators. Functions are suited for stateless computation, while Classes support lifecycle management (@modal.enter() to load the model and @modal.exit() to clean up resources). When you call .remote(), the code runs inside containers managed by Modal, with automatic scaling from zero to thousands of containers.

2. GPU Resource Scheduling

Offer a variety of GPU options, including T4 (cost-effective), L40S (best value for inference), A100 (general-purpose training), H100 (high-performance computing), and H200/B200 (the latest flagship). Support GPU fallback chains (e.g., gpu=["H100", "A100-80GB"]), automatically degrading to a backup GPU when resources are scarce, ensuring your tasks don’t fail due to a specific GPU being unavailable.

3. Container Images and Dependency Management

Define container images entirely using Python code. It’s recommended to use uv for fast package installation. Support system package installation (.apt_install()), executing commands at build time (.run_commands()), and adding local modules (.add_local_python_source()). You can run Python functions to pre-download model weights or datasets; these files are cached into image layers, avoiding repeated downloads on every cold start.

Frequently Asked Questions

Is Modal free? How is it billed?

Modal provides a monthly free tier of $30, enough to try most features. Billing follows a pay-as-you-go model: CPU, memory, and storage are billed based on actual usage and the configured maximum, while GPUs are billed per second. You can check detailed pricing at modal.com/pricing. Rates vary significantly across different GPU types (e.g., T4 is cheaper, H100 is more expensive).

How is Modal different from traditional cloud servers (e.g., AWS EC2)?

Traditional cloud servers require manual setup, configuration, and maintenance, while Modal is fully serverless. You don’t need to manage servers, Docker, or load balancers—just write the code (in Python) that you want to run, and Modal handles scaling, fault tolerance, and infrastructure management. In addition, Modal’s cold start time is typically sub-second, far faster than traditional cloud server startup times.

How do I save and access large model weights in Modal?

Use Modal Volumes for persistent storage. Volumes are optimized for workloads characterized by “write once, read many times” (such as model weights and datasets). Create a volume with modal.Volume.from_name(), then mount it to a specified path inside your function (e.g., volumes={"/data": vol}). You can write files to the volume via the CLI (modal volume put) or through code. The data persists across function calls and can be shared across multiple containers.