YAML config

This is the workflow generated from your input YAML config, which helps you to understand how microservices work together in GNES.

Workflow
graph TD Frontend000(Frontend)-- push/pull -->Preprocessor100(Preprocessor0) Frontend000(Frontend)-- push/pull -->Preprocessor101(Preprocessor1) Preprocessor100(Preprocessor0)-- push/pull -->Router400((Router)) Preprocessor101(Preprocessor1)-- push/pull -->Router400((Router)) Router400((Router))-- push/pull -->Encoder200(Encoder0) Router400((Router))-- push/pull -->Encoder201(Encoder1) Router400((Router))-- push/pull -->Encoder202(Encoder2) Encoder200(Encoder0)-- push/pull -->Router500((Router)) Encoder201(Encoder1)-- push/pull -->Router500((Router)) Encoder202(Encoder2)-- push/pull -->Router500((Router)) Router500((Router))-- pub/sub -->Router600((Router0)) Router500((Router))-- pub/sub -->Router610((Router1)) Router600((Router0))-- push/pull -->Indexer300(Indexer00) Router600((Router0))-- push/pull -->Indexer301(Indexer01) Router600((Router0))-- push/pull -->Indexer302(Indexer02) Router600((Router0))-- push/pull -->Indexer303(Indexer03) Router610((Router1))-- push/pull -->Indexer310(Indexer10) Router610((Router1))-- push/pull -->Indexer311(Indexer11) Router610((Router1))-- push/pull -->Indexer312(Indexer12) Indexer300(Indexer00)-- push/pull -->Router700((Router)) Indexer301(Indexer01)-- push/pull -->Router700((Router)) Indexer302(Indexer02)-- push/pull -->Router700((Router)) Indexer303(Indexer03)-- push/pull -->Router700((Router)) Indexer310(Indexer10)-- push/pull -->Router700((Router)) Indexer311(Indexer11)-- push/pull -->Router700((Router)) Indexer312(Indexer12)-- push/pull -->Router700((Router)) Router700((Router))-- push/pull -->Frontend000(Frontend) classDef FrontendCLS fill:#FFE0E0,stroke:#FFE0E0,stroke-width:1px; classDef EncoderCLS fill:#FFDAAF,stroke:#FFDAAF,stroke-width:1px; classDef IndexerCLS fill:#FFFBC1,stroke:#FFFBC1,stroke-width:1px; classDef RouterCLS fill:#C9E8D2,stroke:#C9E8D2,stroke-width:1px; classDef PreprocessorCLS fill:#CEEEEF,stroke:#CEEEEF,stroke-width:1px; class Frontend000 FrontendCLS; class Preprocessor101,Preprocessor100 PreprocessorCLS; class Router600,Router610,Router400,Router500,Router700 RouterCLS; class Encoder200,Encoder201,Encoder202 EncoderCLS; class Indexer302,Indexer301,Indexer311,Indexer312,Indexer303,Indexer310,Indexer300 IndexerCLS;

This is a Bash script generated from your YAML config. You can use it to start a GNES server on a local machine.


1. Install GNES via pip install gnes
2. Create a new file say run.sh
3. Copy the following content to it and run it via bash ./run.sh.

Shell script
                    
#!/usr/bin/env bash

## Prerequirment of this script
## You need to install GNES locally on this local machine
## pip install gnes

set -e

trap 'kill $(jobs -p)' EXIT

printf "starting service \e[1;33mFrontend\e[0m with \e[1;33m1\e[0m replicas...\n"
gnes frontend --grpc_port 5566 --port_out 58501 --socket_out PUSH_BIND --socket_in PULL_BIND --port_in 50416  &
printf "starting service \e[1;33mPreprocessor\e[0m with \e[1;33m2\e[0m replicas...\n"
gnes preprocess --port_in 58501 --socket_in PULL_CONNECT --port_out 58922 --socket_out PUSH_CONNECT  &
gnes preprocess --port_in 58501 --socket_in PULL_CONNECT --port_out 58922 --socket_out PUSH_CONNECT  &
printf "starting service \e[1;33mRouter\e[0m with \e[1;33m1\e[0m replicas...\n"
gnes route --yaml_path BaseRouter --socket_in PULL_BIND --socket_out PUSH_BIND --port_in 58922 --port_out 58159  &
printf "starting service \e[1;33mEncoder\e[0m with \e[1;33m3\e[0m replicas...\n"
gnes encode --port_in 58159 --socket_in PULL_CONNECT --port_out 63540 --socket_out PUSH_CONNECT  &
gnes encode --port_in 58159 --socket_in PULL_CONNECT --port_out 63540 --socket_out PUSH_CONNECT  &
gnes encode --port_in 58159 --socket_in PULL_CONNECT --port_out 63540 --socket_out PUSH_CONNECT  &
printf "starting service \e[1;33mRouter\e[0m with \e[1;33m1\e[0m replicas...\n"
gnes route --yaml_path ""!PublishRouter {parameters: {num_part: 2}}"" --socket_in PULL_BIND --socket_out PUB_BIND --port_in 63540 --port_out 57269  &
printf "starting service \e[1;33mRouter\e[0m with \e[1;33m1\e[0m replicas...\n"
gnes route --yaml_path BaseRouter --socket_in SUB_CONNECT --socket_out PUSH_BIND --port_in 57269 --port_out 59858  &
printf "starting service \e[1;33mRouter\e[0m with \e[1;33m1\e[0m replicas...\n"
gnes route --yaml_path BaseRouter --socket_in SUB_CONNECT --socket_out PUSH_BIND --port_in 57269 --port_out 52236  &
printf "starting service \e[1;33mIndexer\e[0m with \e[1;33m4\e[0m replicas...\n"
gnes index --yaml_path indexer-binary.yml --port_in 59858 --socket_in PULL_CONNECT --socket_out PUSH_CONNECT --port_out 56050  &
gnes index --yaml_path indexer-binary.yml --port_in 59858 --socket_in PULL_CONNECT --socket_out PUSH_CONNECT --port_out 56050  &
gnes index --yaml_path indexer-binary.yml --port_in 59858 --socket_in PULL_CONNECT --socket_out PUSH_CONNECT --port_out 56050  &
gnes index --yaml_path indexer-binary.yml --port_in 59858 --socket_in PULL_CONNECT --socket_out PUSH_CONNECT --port_out 56050  &
printf "starting service \e[1;33mIndexer\e[0m with \e[1;33m3\e[0m replicas...\n"
gnes index --yaml_path indexer-fulltext.yml --port_in 52236 --socket_in PULL_CONNECT --socket_out PUSH_CONNECT --port_out 56050  &
gnes index --yaml_path indexer-fulltext.yml --port_in 52236 --socket_in PULL_CONNECT --socket_out PUSH_CONNECT --port_out 56050  &
gnes index --yaml_path indexer-fulltext.yml --port_in 52236 --socket_in PULL_CONNECT --socket_out PUSH_CONNECT --port_out 56050  &
printf "starting service \e[1;33mRouter\e[0m with \e[1;33m1\e[0m replicas...\n"
gnes route --yaml_path BaseReduceRouter --socket_in PULL_BIND --socket_out PUSH_CONNECT --port_in 56050 --port_out 50416  &

wait
                    
                

This is a docker-compose YAML file generated from your YAML config. You can use it to start a Docker Swarm distributed on multiple machines.


1. Install Docker and Docker Swarm
2. Create a new file say my-gnes.yml
3. Copy the following content to it
4. Run docker stack deploy --compose-file my-gnes.yml gnes-531.

Docker-Swarm/Compose config
                    
version: '3.4'
services:
  Frontend00:
    image: gnes/gnes:latest-alpine
    command: frontend --grpc_port 5566 --port_out 58501 --socket_out PUSH_BIND --socket_in
      PULL_BIND --port_in 50416
    ports:
    - 5566:5566
  Preprocessor10:
    image: gnes/gnes:latest-alpine
    command: preprocess --port_in 58501 --socket_in PULL_CONNECT --port_out 58922
      --socket_out PUSH_CONNECT --host_out Router20 --host_in Frontend00
    deploy:
      replicas: 2
      restart_policy:
        condition: on-failure
        max_attempts: 3
  Router20:
    image: gnes/gnes:latest-alpine
    command: route --socket_in PULL_BIND --socket_out PUSH_BIND --port_in 58922 --port_out
      58159 --yaml_path BaseRouter
  Encoder30:
    image: gnes/gnes:latest-alpine
    command: encode --port_in 58159 --socket_in PULL_CONNECT --port_out 63540 --socket_out
      PUSH_CONNECT --host_out Router40 --host_in Router20
    deploy:
      replicas: 3
      restart_policy:
        condition: on-failure
        max_attempts: 3
  Router40:
    image: gnes/gnes:latest-alpine
    command: 'route --socket_in PULL_BIND --socket_out PUB_BIND --port_in 63540 --port_out
      57269 --yaml_path "!PublishRouter {parameters: {num_part: 2}}"'
  Router50:
    image: gnes/gnes:latest-alpine
    command: route --socket_in SUB_CONNECT --socket_out PUSH_BIND --port_in 57269
      --port_out 59858 --yaml_path BaseRouter --host_in Router40
  Router51:
    image: gnes/gnes:latest-alpine
    command: route --socket_in SUB_CONNECT --socket_out PUSH_BIND --port_in 57269
      --port_out 52236 --yaml_path BaseRouter --host_in Router40
  Indexer60:
    image: gnes/gnes:latest-alpine
    command: index --port_in 59858 --socket_in PULL_CONNECT --socket_out PUSH_CONNECT
      --port_out 56050 --yaml_path /Indexer60_yaml --host_out Router70 --host_in Router50
    deploy:
      replicas: 4
      restart_policy:
        condition: on-failure
        max_attempts: 3
    configs:
    - Indexer60_yaml
  Indexer61:
    image: gnes/gnes:latest-alpine
    command: index --port_in 52236 --socket_in PULL_CONNECT --socket_out PUSH_CONNECT
      --port_out 56050 --yaml_path /Indexer61_yaml --host_out Router70 --host_in Router51
    deploy:
      replicas: 3
      restart_policy:
        condition: on-failure
        max_attempts: 3
    configs:
    - Indexer61_yaml
  Router70:
    image: gnes/gnes:latest-alpine
    command: route --socket_in PULL_BIND --socket_out PUSH_CONNECT --port_in 56050
      --port_out 50416 --yaml_path BaseReduceRouter --host_out Frontend00
volumes: {}
networks:
  gnes-net:
    driver: overlay
    attachable: true
configs:
  Indexer60_yaml:
    file: indexer-binary.yml
  Indexer61_yaml:
    file: indexer-fulltext.yml
                    
                

The generation of Kubenetes config is currently under development.

Docker-Swarm/Docker-compose config
                    
{{gnes-k8s}}