YAML config

This is the workflow generated from your input YAML config, which helps you to understand how microservices work together in GNES.

Workflow
graph TD Frontend000(Frontend)-- push/pull -->Preprocessor100(Preprocessor0) Frontend000(Frontend)-- push/pull -->Preprocessor101(Preprocessor1) Preprocessor100(Preprocessor0)-- push/pull -->Router400((Router)) Preprocessor101(Preprocessor1)-- push/pull -->Router400((Router)) Router400((Router))-- push/pull -->Encoder200(Encoder0) Router400((Router))-- push/pull -->Encoder201(Encoder1) Router400((Router))-- push/pull -->Encoder202(Encoder2) Encoder200(Encoder0)-- push/pull -->Router500((Router)) Encoder201(Encoder1)-- push/pull -->Router500((Router)) Encoder202(Encoder2)-- push/pull -->Router500((Router)) Router500((Router))-- pub/sub -->Router600((Router0)) Router500((Router))-- pub/sub -->Router610((Router1)) Router600((Router0))-- push/pull -->Indexer300(Indexer00) Router600((Router0))-- push/pull -->Indexer301(Indexer01) Router600((Router0))-- push/pull -->Indexer302(Indexer02) Router600((Router0))-- push/pull -->Indexer303(Indexer03) Router610((Router1))-- push/pull -->Indexer310(Indexer10) Router610((Router1))-- push/pull -->Indexer311(Indexer11) Router610((Router1))-- push/pull -->Indexer312(Indexer12) Indexer300(Indexer00)-- push/pull -->Router700((Router)) Indexer301(Indexer01)-- push/pull -->Router700((Router)) Indexer302(Indexer02)-- push/pull -->Router700((Router)) Indexer303(Indexer03)-- push/pull -->Router700((Router)) Indexer310(Indexer10)-- push/pull -->Router700((Router)) Indexer311(Indexer11)-- push/pull -->Router700((Router)) Indexer312(Indexer12)-- push/pull -->Router700((Router)) Router700((Router))-- push/pull -->Frontend000(Frontend) classDef FrontendCLS fill:#FFE0E0,stroke:#FFE0E0,stroke-width:1px; classDef EncoderCLS fill:#FFDAAF,stroke:#FFDAAF,stroke-width:1px; classDef IndexerCLS fill:#FFFBC1,stroke:#FFFBC1,stroke-width:1px; classDef RouterCLS fill:#C9E8D2,stroke:#C9E8D2,stroke-width:1px; classDef PreprocessorCLS fill:#CEEEEF,stroke:#CEEEEF,stroke-width:1px; class Frontend000 FrontendCLS; class Preprocessor100,Preprocessor101 PreprocessorCLS; class Router610,Router500,Router600,Router700,Router400 RouterCLS; class Encoder200,Encoder201,Encoder202 EncoderCLS; class Indexer311,Indexer303,Indexer301,Indexer312,Indexer302,Indexer300,Indexer310 IndexerCLS;

This is a Bash script generated from your YAML config. You can use it to start a GNES server on a local machine.


1. Install GNES via pip install gnes
2. Create a new file say run.sh
3. Copy the following content to it and run it via bash ./run.sh.

Shell script
                    
#!/usr/bin/env bash

## Prerequirment of this script
## You need to install GNES locally on this local machine
## pip install gnes

set -e

trap 'kill $(jobs -p)' EXIT

printf "starting service \e[1;33mFrontend\e[0m with \e[1;33m1\e[0m replicas...\n"
gnes frontend --grpc_port 5566 --port_out 63120 --socket_out PUSH_BIND --socket_in PULL_BIND --port_in 63720  &
printf "starting service \e[1;33mPreprocessor\e[0m with \e[1;33m2\e[0m replicas...\n"
gnes preprocess --port_in 63120 --socket_in PULL_CONNECT --port_out 63273 --socket_out PUSH_CONNECT  &
gnes preprocess --port_in 63120 --socket_in PULL_CONNECT --port_out 63273 --socket_out PUSH_CONNECT  &
printf "starting service \e[1;33mRouter\e[0m with \e[1;33m1\e[0m replicas...\n"
gnes route --yaml_path BaseRouter --socket_in PULL_BIND --socket_out PUSH_BIND --port_in 63273 --port_out 61353  &
printf "starting service \e[1;33mEncoder\e[0m with \e[1;33m3\e[0m replicas...\n"
gnes encode --port_in 61353 --socket_in PULL_CONNECT --port_out 58564 --socket_out PUSH_CONNECT  &
gnes encode --port_in 61353 --socket_in PULL_CONNECT --port_out 58564 --socket_out PUSH_CONNECT  &
gnes encode --port_in 61353 --socket_in PULL_CONNECT --port_out 58564 --socket_out PUSH_CONNECT  &
printf "starting service \e[1;33mRouter\e[0m with \e[1;33m1\e[0m replicas...\n"
gnes route --yaml_path ""!PublishRouter {parameters: {num_part: 2}}"" --socket_in PULL_BIND --socket_out PUB_BIND --port_in 58564 --port_out 57223  &
printf "starting service \e[1;33mRouter\e[0m with \e[1;33m1\e[0m replicas...\n"
gnes route --yaml_path BaseRouter --socket_in SUB_CONNECT --socket_out PUSH_BIND --port_in 57223 --port_out 55202  &
printf "starting service \e[1;33mRouter\e[0m with \e[1;33m1\e[0m replicas...\n"
gnes route --yaml_path BaseRouter --socket_in SUB_CONNECT --socket_out PUSH_BIND --port_in 57223 --port_out 63644  &
printf "starting service \e[1;33mIndexer\e[0m with \e[1;33m4\e[0m replicas...\n"
gnes index --yaml_path indexer-binary.yml --port_in 55202 --socket_in PULL_CONNECT --socket_out PUSH_CONNECT --port_out 55866  &
gnes index --yaml_path indexer-binary.yml --port_in 55202 --socket_in PULL_CONNECT --socket_out PUSH_CONNECT --port_out 55866  &
gnes index --yaml_path indexer-binary.yml --port_in 55202 --socket_in PULL_CONNECT --socket_out PUSH_CONNECT --port_out 55866  &
gnes index --yaml_path indexer-binary.yml --port_in 55202 --socket_in PULL_CONNECT --socket_out PUSH_CONNECT --port_out 55866  &
printf "starting service \e[1;33mIndexer\e[0m with \e[1;33m3\e[0m replicas...\n"
gnes index --yaml_path indexer-fulltext.yml --port_in 63644 --socket_in PULL_CONNECT --socket_out PUSH_CONNECT --port_out 55866  &
gnes index --yaml_path indexer-fulltext.yml --port_in 63644 --socket_in PULL_CONNECT --socket_out PUSH_CONNECT --port_out 55866  &
gnes index --yaml_path indexer-fulltext.yml --port_in 63644 --socket_in PULL_CONNECT --socket_out PUSH_CONNECT --port_out 55866  &
printf "starting service \e[1;33mRouter\e[0m with \e[1;33m1\e[0m replicas...\n"
gnes route --yaml_path BaseReduceRouter --socket_in PULL_BIND --socket_out PUSH_CONNECT --port_in 55866 --port_out 63720  &

wait
                    
                

This is a docker-compose YAML file generated from your YAML config. You can use it to start a Docker Swarm distributed on multiple machines.


1. Install Docker and Docker Swarm
2. Create a new file say my-gnes.yml
3. Copy the following content to it
4. Run docker stack deploy --compose-file my-gnes.yml gnes-531.

Docker-Swarm/Compose config
                    
version: '3.4'
services:
  Frontend00:
    image: gnes/gnes:latest-alpine
    command: frontend --grpc_port 5566 --port_out 63120 --socket_out PUSH_BIND --socket_in
      PULL_BIND --port_in 63720
    ports:
    - 5566:5566
  Preprocessor10:
    image: gnes/gnes:latest-alpine
    command: preprocess --port_in 63120 --socket_in PULL_CONNECT --port_out 63273
      --socket_out PUSH_CONNECT --host_out Router20 --host_in Frontend00
    deploy:
      replicas: 2
      restart_policy:
        condition: on-failure
        max_attempts: 3
  Router20:
    image: gnes/gnes:latest-alpine
    command: route --socket_in PULL_BIND --socket_out PUSH_BIND --port_in 63273 --port_out
      61353 --yaml_path BaseRouter
  Encoder30:
    image: gnes/gnes:latest-alpine
    command: encode --port_in 61353 --socket_in PULL_CONNECT --port_out 58564 --socket_out
      PUSH_CONNECT --host_out Router40 --host_in Router20
    deploy:
      replicas: 3
      restart_policy:
        condition: on-failure
        max_attempts: 3
  Router40:
    image: gnes/gnes:latest-alpine
    command: 'route --socket_in PULL_BIND --socket_out PUB_BIND --port_in 58564 --port_out
      57223 --yaml_path "!PublishRouter {parameters: {num_part: 2}}"'
  Router50:
    image: gnes/gnes:latest-alpine
    command: route --socket_in SUB_CONNECT --socket_out PUSH_BIND --port_in 57223
      --port_out 55202 --yaml_path BaseRouter --host_in Router40
  Router51:
    image: gnes/gnes:latest-alpine
    command: route --socket_in SUB_CONNECT --socket_out PUSH_BIND --port_in 57223
      --port_out 63644 --yaml_path BaseRouter --host_in Router40
  Indexer60:
    image: gnes/gnes:latest-alpine
    command: index --port_in 55202 --socket_in PULL_CONNECT --socket_out PUSH_CONNECT
      --port_out 55866 --yaml_path /Indexer60_yaml --host_out Router70 --host_in Router50
    deploy:
      replicas: 4
      restart_policy:
        condition: on-failure
        max_attempts: 3
    configs:
    - Indexer60_yaml
  Indexer61:
    image: gnes/gnes:latest-alpine
    command: index --port_in 63644 --socket_in PULL_CONNECT --socket_out PUSH_CONNECT
      --port_out 55866 --yaml_path /Indexer61_yaml --host_out Router70 --host_in Router51
    deploy:
      replicas: 3
      restart_policy:
        condition: on-failure
        max_attempts: 3
    configs:
    - Indexer61_yaml
  Router70:
    image: gnes/gnes:latest-alpine
    command: route --socket_in PULL_BIND --socket_out PUSH_CONNECT --port_in 55866
      --port_out 63720 --yaml_path BaseReduceRouter --host_out Frontend00
volumes: {}
networks:
  gnes-net:
    driver: overlay
    attachable: true
configs:
  Indexer60_yaml:
    file: indexer-binary.yml
  Indexer61_yaml:
    file: indexer-fulltext.yml
                    
                

The generation of Kubenetes config is currently under development.

Docker-Swarm/Docker-compose config
                    
{{gnes-k8s}}