Deployment#

Important

A Deployment is part of the orchestration layer Orchestration. Be sure to read up on that too!

A Deployment orchestrates a single Executor to accomplish a task. Documents are processed by Executors.

You can think of a Deployment as an interface to configure and launch your microservice architecture, while the heavy lifting is done by the service itself.

Why use a Deployment?#

Once you’ve learned about Documents, DocLists and Executors, you can split a big task into small independent modules and services.

Deployments let you scale these Executors independently to match your requirements.
Deployments let you easily use other cloud-native orchestrators, such as Kubernetes, to manage your service.

Create#

The most trivial Deployment is an empty one. It can be defined in Python or from a YAML file:

Python

from jina import Deployment

dep = Deployment()

YAML

jtype: Deployment

For production, you should define your Deployments with YAML. This is because YAML files are independent of the Python logic code and easier to maintain.

Minimum working example#

Pythonic style

from jina import Deployment, Executor, requests
from docarray import DocList, BaseDoc


class MyExecutor(Executor):
    @requests(on='/bar')
    def foo(self, docs: DocList[BaseDoc], **kwargs) -> DocList[BaseDoc]:
        print(docs)


dep = Deployment(name='myexec1', uses=MyExecutor)

with dep:
    dep.post(on='/bar', inputs=BaseDoc(), return_type=DocList[BaseDoc], on_done=print)

Deployment-as-a-Service style

Server:

from jina import Deployment, Executor, requests
from docarray import DocList, BaseDoc


class MyExecutor(Executor):
    @requests(on='/bar')
    def foo(self, docs: DocList[BaseDoc], **kwargs) -> DocList[BaseDoc]:
        print(docs)


dep = Deployment(port=12345, name='myexec1', uses=MyExecutor)

with dep:
    dep.block()

Client:

from jina import Client
from docarray import DocList, BaseDoc

c = Client(port=12345)
c.post(on='/bar', inputs=BaseDoc(), return_type=DocList[BaseDoc], on_done=print)

Load from YAML

deployment.yml:

jtype: Deployment
name: myexec1
uses: FooExecutor
py_modules: exec.py

exec.py:

from jina import Deployment, Executor, requests
from docarray import DocList, BaseDoc
from docarray.documents import TextDoc
 
class FooExecutor(Executor):
    @requests
    def foo(self, docs: DocList[TextDoc], **kwargs) -> DocList[TextDoc]:
        for doc in docs:
            doc.text = 'foo was here'
        docs.summary()
        return docs

from jina import Deployment
from docarray import DocList, BaseDoc
from docarray.documents import TextDoc

dep = Deployment.load_config('deployment.yml')

with dep:
    try:
        dep.post(on='/bar', inputs=TextDoc(), on_done=print)
    except Exception as ex:
        # handle exception
        pass

Caution

The statement with dep: starts the Deployment, and exiting the indented with block stops the Deployment, including its Executors. Exceptions raised inside the with dep: block will close the Deployment context manager. If you don’t want this, use a try...except block to surround the statements that could potentially raise an exception.

Convert between Python and YAML#

A Python Deployment definition can easily be converted to/from a YAML definition:

Load from YAML

from jina import Deployment

dep = Deployment.load_config('flow.yml')

Export to YAML

from jina import Deployment

dep = Deployment()

dep.save_config('deployment.yml')

Start and stop#

When a Deployment starts, all the replicated Executors will start as well, making it possible to reach the service through its API.

There are three ways to start a Deployment: In Python, from a YAML file, or from the terminal.

Generally in Python: use Deployment as a context manager.
As an entrypoint from terminal: use Jina CLI <cli> and a Deployment YAML file.
As an entrypoint from Python code: use Deployment as a context manager inside if __name__ == '__main__'
No context manager, manually call start() and close().

General in Python

from jina import Deployment

dep = Deployment()

with dep:
    pass

The statement with dep: starts the Deployment, and exiting the indented with block stops the Deployment, including its Executor.

Jina CLI entrypoint

jina deployment --uses deployment.yml

Python entrypoint

from jina import Deployment

dep = Deployment()

if __name__ == '__main__':
    with dep:
        pass

The statement with dep: starts the Deployment, and exiting the indented with block stops the Deployment, including its Executor.

Python no context manager

from jina import Deployment

dep = Deployment()

dep.start()

dep.close()

Your addresses and entrypoints can be found in the output. When you enable more features such as monitoring, HTTP gateway, TLS encryption, this display expands to contain more information.

Set multiprocessing `spawn`#

Some corner cases require forcing a spawn start method for multiprocessing, for example if you encounter “Cannot re-initialize CUDA in forked subprocess”.

You can use JINA_MP_START_METHOD=spawn before starting the Python script to enable this.

JINA_MP_START_METHOD=spawn python app.py

Caution

In case you set JINA_MP_START_METHOD=spawn, make sure to use Flow as a context manager inside if __name__ == '__main__'. The script entrypoint (starting the flow) needs to be protected when using spawn start method.

Hint

There’s no need to set this for Windows, as it only supports spawn method for multiprocessing.

Serve#

Serve forever#

In most scenarios, a Deployment should remain reachable for prolonged periods of time. This can be achieved from the terminal:

Python

from jina import Deployment

dep = Deployment()

with dep:
    dep.block()

YAML

jina deployment --uses deployment.yml

The .block() method blocks the execution of the current thread or process, enabling external clients to access the Deployment.

In this case, the Deployment can be stopped by interrupting the thread or process.

Serve until an event#

Alternatively, a multiprocessing or threading Event object can be passed to .block(), which stops the Deployment once set.

from jina import Deployment
import threading


def start_deployment(stop_event):
    """start a blocking Deployment."""
    dep = Deployment()
    
    with dep:
        dep.block(stop_event=stop_event)


e = threading.Event()  # create new Event

t = threading.Thread(name='Blocked-Flow', target=start_flow, args=(e,))
t.start()  # start Deployment in new Thread

# do some stuff

e.set()  # set event and stop (unblock) the Deployment

Export#

A Deployment YAML can be exported as a Docker Compose YAML or Kubernetes YAML bundle.

Docker Compose#

Python

from jina import Deployment

dep = Deployment()
dep.to_docker_compose_yaml()

Terminal

jina export docker-compose deployment.yml docker-compose.yml 

This will generate a single docker-compose.yml file.

For advanced utilization of Docker Compose with Jina, refer to How to

Kubernetes#

Python

from jina import Deployment

dep = Deployment
dep.to_kubernetes_yaml('dep_k8s_configuration')

Terminal

jina export kubernetes deployment.yml ./my-k8s 

The generated folder can be used directly with kubectl to deploy the Deployment to an existing Kubernetes cluster.

For advanced utilisation of Kubernetes with Jina please refer to How to

Tip

Based on your local Jina version, Executor Hub may rebuild the Docker image during the YAML generation process. If you do not wish to rebuild the image, set the environment variable JINA_HUB_NO_IMAGE_REBUILD.

Tip

If an Executor requires volumes to be mapped to persist data, Jina will create a StatefulSet for that Executor instead of a Deployment. You can control the access mode, storage class name and capacity of the attached Persistent Volume Claim by using Jina environment variables
JINA_K8S_ACCESS_MODES, JINA_K8S_STORAGE_CLASS_NAME and JINA_K8S_STORAGE_CAPACITY. Only the first volume will be considered to be mounted.

Logging#

The default jina.logging.logger.JinaLogger uses rich console logging that writes to the system console. The log_config argument can be used to pass in a string of the pre-configured logging configuration names in Jina or the absolute YAML file path of the custom logging configuration. For most cases, the default logging configuration sufficiently covers local, Docker and Kubernetes environments.

Custom logging handlers can be configured by following the Python official Logging Cookbook examples. An example custom logging configuration file defined in a YAML file logging.json.yml is:

handlers:
  - StreamHandler
level: INFO
configs:
  StreamHandler:
    format: '%(asctime)s:{name:>15}@%(process)2d[%(levelname).1s]:%(message)s'
    formatter: JsonFormatter

The logging configuration can be used as follows:

Python

from jina import Deployment

dep = Deployment(log_config='./logging.json.yml')

YAML

jtype: Deployment
with:
    log_config: './logging.json.yml'

Supported protocols#

A Deployment can be used to deploy an Executor and serve it using gRPC or HTTP protocol, or a composition of them.

gRPC protocol#

gRPC is the default protocol used by a Deployment to expose Executors to the outside world, and is used to communicate between the Gateway and an Executor inside a Flow.

HTTP protocol#

HTTP can be used for a stand-alone Deployment (without being part of a Flow), which allows external services to connect via REST.

from jina import Deployment, Executor, requests
from docarray import DocList
from docarray.documents import TextDoc
 
class MyExec(Executor):
    @requests
    def foo(self, docs: DocList[TextDoc], **kwargs) -> DocList[TextDoc]:
        for doc in docs:
            doc.text = 'foo was here'
        docs.summary()
        return docs

dep = Deployment(protocol='http', port=12345, uses=MyExec)

with dep:
    dep.block()

This will make it available at port 12345 and you can get the OpenAPI schema for the service.

../../../_images/http-deployment-swagger.png

Composite protocol#

A Deployment can also deploy an Executor and serve it with a combination of gRPC and HTTP protocols.

from jina import Deployment, Executor, requests
from docarray import DocList
from docarray.documents import TextDoc
 
class MyExec(Executor):
    @requests
    def foo(self, docs: DocList[TextDoc], **kwargs) -> DocList[TextDoc]:
        for doc in docs:
            doc.text = 'foo was here'
        docs.summary()
        return docs


dep = Deployment(protocol=['grpc', 'http'], port=[12345, 12346], uses=MyExec)

with dep:
    dep.block()

This will make the Deployment reachable via gRPC and HTTP simultaneously.

Methods#

The most important methods of the Deployment object are the following:

Method	Description
`start()`	Starts the Deployment. This will start all its Executors and check if they are ready to be used.
`close()`	Stops and closes the Deployment. This will stop and shutdown all its Executors.
`with` context manager	Uses the Deployment as a context manager. It will automatically start and stop your Deployment.
`post()`	Sends requests to the Deployment API.
`block()`	Blocks execution until the program is terminated. This is useful to keep the Deployment alive so it can be used from other places (clients, etc).
`to_docker_compose_yaml()`	Generates a Docker-Compose file listing all Executors as services.
`to_kubernetes_yaml()`	Generates Kubernetes configuration files in `<output_directory>`. Based on your local Jina version, Executor Hub may rebuild the Docker image during the YAML generation process. If you do not wish to rebuild the image, set the environment variable `JINA_HUB_NO_IMAGE_REBUILD`.
`is_deployment_ready()`	Check if the Deployment is ready to process requests. Returns a boolean indicating the readiness.

Deployment#

Why use a Deployment?#

Create#

Minimum working example#

Convert between Python and YAML#

Start and stop#

Set multiprocessing spawn#

Serve#

Serve forever#

Serve until an event#

Export#

Docker Compose#

Kubernetes#

Logging#

Supported protocols#

gRPC protocol#

HTTP protocol#

Composite protocol#

Methods#

Set multiprocessing `spawn`#