Modal Functions

Modal Functions execute Python code using highly scalable serverless cloud compute.

Preparing a function to run on Modal is as simple as defining an App and registering the function using the @app.function() decorator:

app = modal.App("basic-function")

@app.function()
def f(x: int, exp: int) -> int:
    return x**exp

The wrapped function becomes a modal.Function that can be called from a local script or deployed and invoked on demand from other applications as if it were part of their codebase.

When you invoke the Function, Modal handles all of the operational details: booting a container, routing your inputs, and propagating any exceptions. If you don’t send more inputs, the Function will automatically scale to zero so that it incurs no ongoing cost. Information about the Function invocation, any logs it produced, and a rich set of container metrics are automatically captured and presented across a number of observability surfaces.

Configuring the Function runtime 

The Function runtime is configured via arguments to the @app.function() decorator. Everything about the container environment and the resources available to the Function can be defined within your Python codebase, without reference to external configuration files.

Functions receive a baseline allotment of CPU and memory. Explicit resource configuration is not necessary for simple tasks, because Functions can opportunistically burst above this baseline when needed. Heavier jobs can be provisioned with additional resources to guarantee availability:

@app.function(cpu=16, memory=32768)  # 16 physical cores, 32 GiB of RAM
def f():
    ...

Functions can also be provisioned with one or more GPUs:

@app.function(gpu="H200:8")
def f():
    ...

Functions execute within arbitrary container environments, as defined by the Function’s Image. Each Function in the App can have its own Image. Images can include resources including Python libraries from PyPI or private repositories, binary dependencies like FFmpeg or OpenCV, and data copied from your local system:

image  = (
    modal.Image.debian_slim()
    .uv_sync()
    .apt_install("ffmpeg")
    .add_local_dir("data", "/data")
)

@app.function(image=image)
def f():
    ...

If the Function is provisioned with a GPU, the CUDA drivers are automatically included.

Modal includes the Function’s source in the container by default. Depending on the project structure, this will be either the script file or entire package where the Function’s implementation is defined. As a consequence, Functions do not need to be self-contained and can reference other resources in their module.

Larger datasets, such as model weights, can be mounted into the container using a Modal Volume or CloudBucketMount:

vol = modal.Volume.from_name("model-weights")

@app.function(volumes={"/models": vol})
def f():
    ...

Environment variables can be defined in the container runtime by passing them as secure Secrets or by setting them directly:

api_key = modal.Secret.from_name("api-key")

@app.function(secrets=[api_key], env={"LOG_LEVEL": "info"})
def f():
    ...

Function invocation 

Modal Functions are called by one of their invocation methods, such as f.remote() or f.spawn(). When referenced by another Function or local entrypoint in the same App, a Function can be invoked directly:

@app.function()
def f() -> str:
    return "Hello from a Modal container"

@app.function()
def g() -> str:
    return f.remote()

@app.local_entrypoint()
def main():
    print(g.remote())

Functions can be invoked from another App or from outside of Modal after a lookup using the App and Function names:

f = modal.Function.from_name("prod-app", "f")
result = f.remote()

Remote lookups and invocations can also be performed via our JavaScript and Go SDKs, allowing you to execute code that leverages Python’s AI ecosystem from within applications written in other languages:

const f = await modal.functions.fromName("prod-app", "f");
result = await f.remote();

f, _ := mc.Functions.FromName(ctx, "prod-app", "f", nil)
result, err := f.Remote(ctx, nil, nil)

Applying one of the Web Function decorators assigns a URL for the Function and allows you to invoke it from anywhere via HTTP:

image = modal.Image.debian_slim().uv_pip_install("fastapi[standard]")

@app.function(image=image)
@modal.fastapi_endpoint()
def f() -> dict[str, str]:
    return {"message": "Hello from a Modal container"}

Note that Web Functions are open to the internet by default, but they can optionally require authentication via Proxy Tokens.

Web Functions are designed for conveniently exposing simple Python functions as web services; use Modal’s Server primitive instead for high concurrency or latency-sensitive applications.

Functions can also be automatically invoked on a schedule, akin to a cron job:

@app.function(schedule=modal.Cron("0 6 * * *", timezone="America/New_York"))
def f():
    ...

Execution semantics 

Modal Functions abstract several principles of reliable cloud compute orchestration to present an input/output interface that looks like a local Python function call.

Function invocations are automatically authenticated via your Modal token/secret credentials and authorized per your RBAC configuration. The Function implementation does not need to perform access control.

Modal is responsible for scheduling containers and routing your inputs to them. By default, Function containers can start anywhere in our global fleet, which maximizes availability and minimizes scheduling latency. To constrain container scheduling, e.g. for compliance, configure the compute and routing regions:

@app.function(region="eu", routing_region="eu-west")
def f():
    ...

Note that compute region selection incurs a pricing multiplier; routing region selection does not. Region selection also limits the pool of compute, especially when combined with specific GPUs or large resource requests, which can impact scheduling latency.

Because container scheduling is reactive to input load, a container may not be available at the moment of invocation. Inputs will queue in Modal’s I/O system until they can be distributed to available containers. If inputs are enqueued too quickly or the queue fills up, they will be rejected with a ResourceExhaustedError. For batch workloads, prefer the durable f.spawn() method, which supports higher invocation rates and substantially deeper input queues.

Modal applies an input timeout to each invocation; timeouts do not need to be set in the calling context. Timeouts are short by default (5 minutes), but they can be extended up to 24 hours for long-running processes like model training:

@app.function(timeout=86400)  # 24 hours
def f():
    ...

Occasionally, containers will fail while executing inputs, e.g. due to preemption or out-of-memory (OOM) errors. Modal automatically retries any inputs that a container was running when it failed. As a consequence, Function implementations should be idempotent. CPU functions can opt for non-preemptibility, although this incurs a pricing multiplier:

@app.function(nonpreemptible=True)
def f():
    ...

Exceptions that originate in the Function’s implementation are not automatically retried, but input retries can be enabled:

@app.function(retries=3)
def f():
    ...

Autoscaling and parallelism 

Modal Functions autoscale by default. Just as the Function automatically boots a container in response to an initial input, it will boot additional containers if further inputs are received while it is busy. Under ongoing load, the autoscaler will manage the container pool (booting containers or scaling them down) to accommodate fluctuating levels of demand.

Functions expose several options to control the autoscaling behavior. Use min_containers or buffer_containers to reduce cold start penalties by keeping additional idle containers running, and set max_containers to limit scaleup under heavy demand:

@app.function(min_containers=1, buffer_containers=1, max_containers=20)
def f():
    ...

After a container finishes handling an input, it is available for reuse. Container reuse reduces average latency, because subsequent inputs will be handled immediately instead of waiting for a new container to boot. As load decreases, Modal will gradually scale down containers that are idle, and Functions will eventually scale to zero if inputs cease altogether. The scaledown_window controls the aggressiveness of this behavior:

@app.function(scaledown_window=600)  # Idle for longer to better handle sporadic load patterns
def f():
    ...

While most Function configuration requires a redeployment to change, the autoscaler parameters can be dynamically updated using f.update_autoscaler():

f = modal.Function.from_name("prod-app", "f")
f.update_autoscaler(max_containers=50)  # Override the Function's decorator configuration

Note that any dynamic updates will be reset by a subsequent deployment.

Because Functions autoscale rapidly, they are a good fit for bursty workloads or batch jobs that require fan-out parallelism. The batch-oriented f.map() and f.spawn_map() methods facilitate parallel execution by efficiently pushing an iterable of inputs into Modal:

for result in f.map(inputs):  # Iterate in parallel and handle each result
    ...

f.spawn_map(inputs)  # Higher parallelism with durable semantics for fire-and-forget batch jobs

Parallel execution can also be achieved using concurrency patterns. The f.spawn() method returns a modal.FunctionCall, which acts like a Future:

fc = f.spawn(x)
result = fc.get()

Spawning multiple calls allows them to run in parallel:

fcs = [f.spawn(x_i) for x_i in xs]
results = modal.FunctionCall.gather(*fcs)

Async codebases can also use Modal’s aio interface to apply concurrency patterns with any invocation method:

coros = [f.remote.aio(x_i) for x_i in xs]
results = await asyncio.gather(*coros)

Container lifecycle management 

While Modal containers boot in less than a second, your application logic may require expensive additional setup, such as loading model weights from disk. By structuring the Function’s code as a class and using the @app.cls() decorator, you can separate the startup logic from the input handling:

@app.cls()
class InferenceEngine:
    @modal.enter()
    def setup(self):
        self.model = load_model()

    @modal.method()
    def predict(self, text: str) -> float:
        return self.model.predict(text)

In this example, the method wrapped with the @modal.enter() decorator will run only once, as part of container startup. The container will not be considered “ready” until the startup method or methods complete, and Modal will wait for this event before sending the container any inputs.

A Cls is invoked by “constructing” the class and calling the method decorated with @modal.method(). As with normal Functions, this can be a local reference or a lookup:

result = InferenceEngine().predict.remote(text)  # Refer to a Cls on the same App

InferenceEngine = modal.Cls.from_name("prod-app", "InferenceEngine")
result = InferenceEngine().predict.remote(text)  # Refer to a Cls via a lookup

Structuring your code as a class also lets you define container teardown logic in methods wrapped with the @modal.exit() decorator. This is useful for cleanup operations like gracefully closing connections to databases. The exit handler can also be used to make your application more resilient to container preemption.

Any state written to the self namespace will persist across the calls handled by an individual container, but it will be discarded when the container terminates. State can be shared across containers using Modal’s distributed Dict or Queue primitives.

If the Function produces local state that should not leak across inputs, you can set single_use_containers=True. This causes each container to terminate after handling an input. Note that single-use containers add some latency and cost, since they do not benefit from amortizing container startup over multiple inputs.

Function parametrization 

To write templated container lifecycle logic, add modal.parameter() declarations to the class:

@app.cls()
class InferenceEngine:
    model_name: str = modal.parameter()

    @modal.enter()
    def startup(self):
        self.model = load_model(self.model_name)

    @modal.method()
    def predict(self, input: str) -> float:
        ...

This creates a Parametrized Function. Supply values for the parameters when constructing the Cls in a calling context, which creates a specific instance of the Function:

result = InferenceEngine(model_name="tts-large").predict.remote(text)

Because the parameters apply to the entire container lifecycle, every distinct set of parameter values corresponds to a separate, independently autoscaling container pool. This can also be leveraged to partition a Function’s containers, even when the parameter values are not read at startup. For example, you may wish to process data from different customers in separate containers:

result = PartitionedInferenceEngine(customer_id="c-024").predict.remote(text)

Note that there is a limit on the number of distinct instances each Function can have, so this approach is only suited for partitioning schemes with relatively low cardinality. Prefer using single_use_containers= for container isolation when parameter values would not frequently recur and benefit from container reuse.

Dynamic configuration 

Updating configuration values in the @app.function() decorator requires a redeployment, but it’s also possible to dynamically configure the Function from a call site using f.with_options(). This is useful in cases where specific inputs or parameter values require different resources, such as different GPUs:

result = InferenceEngine(model_name="tts-large").predict.remote(text)

InferenceEngineH200 = InferenceEngine.with_options(gpu="H200")
result = InferenceEngineH200(model_name="tts-xlarge").predict.remote(text)

As with Parametrized Functions (but unlike updates to the autoscaler configuration), each distinct set of dynamic options corresponds to an independent container pool. If dynamically configuring CPU or memory, use a coarse set of values to benefit from container reuse.

Concurrency and batching 

By default, each Function container will handle one input at a time. Functions support two distinct patterns for handling multiple inputs.

Input concurrency, enabled using the @modal.concurrent() decorator, allows Functions to accept multiple inputs and execute them concurrently using either threads or asyncio tasks:

@app.function()
@modal.concurrent(max_inputs=10)
def f(x):
    ...  # Sync implementation; each input runs in its own thread

@app.function()
@modal.concurrent(max_inputs=10)
async def g(x):
    ...  # Async implementation; each input runs on the main thread in an asyncio task

Functions can benefit from input concurrency if they are I/O bound, e.g. because they make network requests or database queries. Some GPU frameworks can also benefit from input concurrency via continuous batching. Input concurrency is less likely to be useful if the Function is CPU bound.

An alternative strategy is dynamic batching, enabled using the @modal.batched() decorator. A batched Function must be defined as accepting a list (or lists) of inputs and returning a list of outputs:

@app.function()
@modal.batched(max_batch_size=4, wait_ms=1000)
def f(x: list[int], y: list[int]) -> list[int]:
    return [x_i + y_i for x_i, y_i in zip(x, y)]

When calling a batched Function, inputs are sent individually, buffered by Modal until the batch size is filled or the wait period elapses, and then processed in a single function call. From the perspective of any individual caller, this looks no different from a normal Function invocation:

xy_sum = f.remote(2, 6)

Dynamic batching is especially useful in cases where you can leverage vectorization via tensor or array frameworks like torch or numpy.