feat: docker/docker-compose/documentation for temporal mode

Author: Klesh Wong

README.md

@@ -245,6 +245,37 @@ To synchronize data periodically, we provide [`lake-cli`](./cmd/lake-cli/README.
 <br>
 
 
+## Temporal Mode
+
+Normally, DevLake would execute pipelines on local machine (we call it `local mode`), it is sufficient most of the time.However, when you have too many pipelines that need to be executed in parallel, it can be problematic, either limited by the horsepower or throughput of a single machine.
+
+`temporal mode` was added to support distributed pipeline execution, you can fire up arbitrary workers on multiple machines to carry out those pipelines in parallel without hitting the single machine limitation.
+
+But, be careful, many API services like JIRA/GITHUB have request rate limit mechanism, collect data in parallel against same API service with same identity would most likely hit the wall.
+
+### How it works
+
+1. DevLake Server and Workers connect to the same temporal server by setting up `TEMPORAL_URL`
+2. DevLake Server sends `pipeline` to temporal server, and one of the Workers would pick it up and execute
+
+
+**IMPORTANT: This feature is in early stage of development, use with cautious**
+
+
+### Temporal Demo
+
+#### Requirements
+
+- [Docker](https://docs.docker.com/get-docker)
+- [docker-compose](https://docs.docker.com/compose/install/)
+- [temporalio](https://github.com/merico-dev/lake/pull/1559)
+
+#### How to setup
+
+1. Clone and fire up  [temporalio](https://github.com/merico-dev/lake/pull/1559) services
+2. Clone this repo, and fire up DevLake with command `docker-compose -f docker-compose-temporal.yml up -d`
+
+
 ## Project Roadmap
 - <a href="https://github.com/merico-dev/lake/wiki/Roadmap-2022" target="_blank">Roadmap 2022</a>: Detailed project roadmaps for 2022.
 - DevLake already supported following data sources:

docker-compose-temporal.yml

@@ -0,0 +1,100 @@
+version: "3"
+services:
+  mysql:
+    image: mysql:8.0.26
+    platform: linux/x86_64
+    volumes:
+      - mysql-storage:/var/lib/mysql
+    restart: always
+    ports:
+      - 127.0.0.1:3306:3306
+    environment:
+      MYSQL_ROOT_PASSWORD: admin
+      MYSQL_DATABASE: lake
+      MYSQL_USER: merico
+      MYSQL_PASSWORD: merico
+
+  grafana:
+    image: mericodev/grafana:latest
+    build:
+      context: grafana/
+    ports:
+      - 3002:3000
+    volumes:
+      - grafana-storage:/var/lib/grafana
+    environment:
+      GF_USERS_ALLOW_SIGN_UP: 'false'
+      GF_DASHBOARDS_JSON_ENABLED: 'true'
+      GF_INSTALL_PLUGINS: grafana-piechart-panel
+      GF_LIVE_ALLOWED_ORIGINS: '*'
+      MYSQL_URL: mysql:3306
+      MYSQL_DATABASE: lake
+      MYSQL_USER: merico
+      MYSQL_PASSWORD: merico
+    restart: always
+    depends_on:
+      - mysql
+
+  devlake:
+    image: mericodev/lake:latest
+    build:
+      context: "."
+      args:
+        HTTPS_PROXY: "${HTTPS_PROXY}"
+        GOPROXY: "${GOPROXY}"
+    ports:
+      - 127.0.0.1:8080:8080
+    restart: always
+    volumes:
+      - ./.env:/app/.env
+    depends_on:
+      - mysql
+    networks:
+      - default
+      - temporal-network
+    environment:
+      - TEMPORAL_URL=temporal:7233
+
+  devlake-worker:
+    image: mericodev/lake:latest
+    build:
+      context: "."
+      args:
+        HTTPS_PROXY: "${HTTPS_PROXY}"
+        GOPROXY: "${GOPROXY}"
+    restart: always
+    volumes:
+      - ./.env:/app/.env
+    depends_on:
+      - mysql
+    command: lake-worker
+    networks:
+      - default
+      - temporal-network
+    environment:
+      - TEMPORAL_URL=temporal:7233
+
+  config-ui:
+    image: mericodev/config-ui:latest
+    build:
+      context: "config-ui"
+    ports:
+      - 127.0.0.1:4000:80
+    env_file:
+      - ./.env
+    #environment:
+      #ADMIN_USER: devlake
+      #ADMIN_PASS: merico
+    depends_on:
+      - devlake
+    networks:
+      - default
+
+volumes:
+  mysql-storage:
+  grafana-storage:
+
+networks:
+  default:
+  temporal-network:
+    external: true