Configuration & Cluster Deployment

Background
Prerequisite
Deploy an Easegress Cluster Step by Step
- Add New Member
YAML Configuration
Configuration and Environment Variables
Configuration tips (optional)
References

Background

When to deploy Easegress as a cluster?

Your traffic is larger than one machine can handle
You want to High-availability and minimize the service downtime
You want to minimize latency on service peaks

It is easy to start multiple Easegress instances to form an Easegress cluster. This tutorial provides instructions on how to create a stand-alone Easegress cluster by starting multiple Easegress instances.

Prerequisite

The following prerequisites are required for a successful deployment of the Easegress cluster.

latest easegress-server and egctl binaries (run make in the root of the repository)
successful creation of an Easegress pipeline (like the Hello World example in the of the repository or any other chapter in tutorial and cookbook.
few machines that are in the same network or otherwise accessible or Docker or other container technology. If you only have one machine, then you could use localhost as the host and modify the ports in the example.

Deploy an Easegress Cluster Step by Step

The goal of this tutorial is to have following infrastructure running Easegress:

4 machines connected
each running Easegress instance
3 of Easegress instances have cluster role primary and one secondary

The difference between primary and secondary cluster roles is that primary persists the cluster state to disk, while secondary does not. The number of secondary Easegress instances can scale up and down, but the number of primary instances should be fixed.

Let’s start by creating three Easegress instances with primary role. Add nodes private IPs to following environment variables:

export HOST1=<host1-IP>
export HOST2=<host2-IP>
export HOST3=<host3-IP>
export CLUSTER=machine-1=http://$HOST1:2380,machine-2=http://$HOST2:2380,machine-3=http://$HOST3:2380

CLUSTER environment variable now contains IP addresses of each member in the cluster. It will be the same for all members.

Set the environment variables to each machine. Start the first instance at the first machine

easegress-server \
  --cluster-name "multi-node-cluster" \
  --cluster-role "primary" \
  --name "machine-1" \
  --api-addr $HOST1:2381 \
  --initial-advertise-peer-urls http://$HOST1:2380 \
  --listen-peer-urls http://$HOST1:2380 \
  --listen-client-urls http://$HOST1:2379 \
  --advertise-client-urls http://$HOST1:2379 \
  --initial-cluster $CLUSTER

Here we define the basic information, like the name of the instance and the name of the cluster. Arguments initial-advertise-peer-urls,listen-peer-urls, listen-client-urls and advertise-client-urls are for communication with other peers (other primary cluster members). You can read more about them at the end of this tutorial, but for now, it’s enough to notice that hostname for machine-1 is $HOST1, which is the IP address of this machine.

Then start the second instance at machine 2

easegress-server \
  --cluster-name "multi-node-cluster" \
  --cluster-role "primary" \
  --name "machine-2" \
  --initial-advertise-peer-urls http://$HOST2:2380 \
  --listen-peer-urls http://$HOST2:2380 \
  --listen-client-urls http://$HOST2:2379 \
  --advertise-client-urls http://$HOST2:2379 \
  --initial-cluster $CLUSTER

and the last machine 3.

easegress-server \
  --cluster-name "multi-node-cluster" \
  --cluster-role "primary" \
  --name "machine-3" \
  --initial-advertise-peer-urls http://$HOST3:2380 \
  --listen-peer-urls http://$HOST3:2380 \
  --listen-client-urls http://$HOST3:2379 \
  --advertise-client-urls http://$HOST3:2379 \
  --initial-cluster $CLUSTER

Now you can list cluster members

egctl --server $HOST1:2381 get member

should print something like

NAME              ROLE            AGE     STATE     API-ADDR     HEARTBEAT
machine-1         primary         10m     Leader    $HOST1:2381  3s ago
machine-2         primary         10m     Follower  $HOST2:2381  4s ago
machine-3         primary         10m     Follower  $HOST3:2381  2s ago

Add New Member

Let’s add one more node with a secondary cluster role this time.

---
title: on machine 4
linkTitle: on machine 4
weight: 1
---
easegress-server \
  --cluster-name "multi-node-cluster" \
  --cluster-role "secondary" \
  --name "machine-4" \
  --primary-listen-peer-urls http://$HOST1:2380 \
  --state-flag "existing"

Here primary-listen-peer-urls tell, where to find a primary cluster members and state-flag with value “existing” means that this cluster was already created.

We can now see also the 4th instance:

egctl --server $HOST1:2381 get member

prints to output:

NAME              ROLE            AGE     STATE     API-ADDR     HEARTBEAT
machine-1         primary         10m     Leader    $HOST1:2381  3s ago
machine-2         primary         10m     Follower  $HOST2:2381  4s ago
machine-3         primary         10m     Follower  $HOST3:2381  2s ago
machine-4         secondary       2m      ""        $HOST1:2381  3s ago

Congratulations, you now have your Easegress instances running! You can now start applying resources to Easegress, like httpserver or pipeline for example.

You can also keep reading this tutorial to know more about YAML configuration of Easegress cluster instances or configuration tips.

YAML Configuration

The examples above use the easegress-server’s command-line flags, but often it is more convenient to define server parameters in a YAML configuration file. For example, store the following YAML to each host machine and change the host addresses accordingly.

---
title: create one yaml file for each host
linkTitle: create one yaml file for each host
weight: 1
---
name: machine-1 # machine-2, machine-3
cluster-name: cluster-test
cluster-role: primary
api-addr: localhost:2381
data-dir: ./data
wal-dir: ""
cpu-profile-file:
memory-profile-file:
log-dir: ./log
debug: false
cluster:
  listen-peer-urls: # change CURRENT-HOST to current host
   - http://<CURRENT-HOST>:2380
  listen-client-urls:
   - http://<CURRENT-HOST>:2379
  advertise-client-urls:
   - http://<CURRENT-HOST>:2379
  initial-advertise-peer-urls:
   - http://<CURRENT-HOST>:2380
  initial-cluster: # initial-cluster is same for every host
   - machine-1: http://<HOST-1>:2380
   - machine-2: http://<HOST-2>:2380
   - machine-3: http://<HOST-3>:2380

Then apply these values on each machine, using config-file command line argument: easegress-server --config-file config.yaml.

The configuration file for adding a new secondary node looks like following:

name: machine-4
cluster-name: cluster-test
cluster-role: secondary
data-dir: ./data
wal-dir: ""
cpu-profile-file:
memory-profile-file:
log-dir: ./log
debug: false
cluster:
  primary-listen-peer-urls:
   - http://$HOST1:2380

Configuration and Environment Variables

In addition to deploying the easegress-server using command-line flags or a YAML file, you can also utilize environment variables. Below are all the available environment variables along with their corresponding flags.

---
title: Human-readable name for this member.
linkTitle: Human-readable name for this member.
weight: 1
---
EASEGRESS_NAME:                --name

---
title: The labels for the instance of Easegress.
linkTitle: The labels for the instance of Easegress.
weight: 1
---
EASEGRESS_LABELS:              --labels

---
title: Load server configuration from a file(yaml format), other command line flags will be ignored if specified.
linkTitle: Load server configuration from a file(yaml format), other command line flags will be ignored if specified.
weight: 1
---
EASEGRESS_CONFIG_FILE:         --config-file

---
title: Force to create a new one-member cluster.
linkTitle: Force to create a new one-member cluster.
weight: 1
---
EASEGRESS_FORCE_NEW_CLUSTER:   --force-new-cluster

---
title: Send an upgrade signal to the server based on the local pid file, then exit. The original server will start a graceful upgrade after signal received.
linkTitle: Send an upgrade signal to the server based on the local pid file, then exit. The original server will start a graceful upgrade after signal received.
weight: 1
---
EASEGRESS_SIGNAL_UPGRADE:      --signal-upgrade

---
title: Use standalone etcd instead of embedded.
linkTitle: Use standalone etcd instead of embedded.
weight: 1
---
EASEGRESS_USE_STANDALONE_ETCD: --use-standalone-etcd

---
title: Human-readable name for the new cluster, ignored while joining an existed cluster.
linkTitle: Human-readable name for the new cluster, ignored while joining an existed cluster.
weight: 1
---
EASEGRESS_CLUSTER_NAME:            --cluster-name

---
title: Cluster role for this member (primary, secondary).
linkTitle: Cluster role for this member (primary, secondary).
weight: 1
---
EASEGRESS_CLUSTER_ROLE:            --cluster-role

---
title: Timeout to handle request in the cluster.
linkTitle: Timeout to handle request in the cluster.
weight: 1
---
EASEGRESS_CLUSTER_REQUEST_TIMEOUT: --cluster-request-timeout

---
title: List of URLs to listen on for cluster client traffic.
linkTitle: List of URLs to listen on for cluster client traffic.
weight: 1
---
EASEGRESS_LISTEN_CLIENT_URLS:          --listen-client-urls

---
title: List of URLs to listen on for cluster peer traffic.
linkTitle: List of URLs to listen on for cluster peer traffic.
weight: 1
---
EASEGRESS_LISTEN_PEER_URLS:            --listen-peer-urls

---
title: List of this member's client URLs to advertise to the rest of the cluster.
linkTitle: List of this member's client URLs to advertise to the rest of the cluster.
weight: 1
---
EASEGRESS_ADVERTISE_CLIENT_URLS:       --advertise-client-urls

---
title: List of this member's peer URLs to advertise to the rest of the cluster.
linkTitle: List of this member's peer URLs to advertise to the rest of the cluster.
weight: 1
---
EASEGRESS_INITIAL_ADVERTISE_PEER_URLS: --initial-advertise-peer-urls

---
title: List of (member name, URL) pairs that will form the cluster. E.g. primary-1=http://localhost:2380.
linkTitle: List of (member name, URL) pairs that will form the cluster. E.g. primary-1=http://localhost:2380.
weight: 1
---
EASEGRESS_INITIAL_CLUSTER:             --initial-cluster

---
title: Cluster state (new, existing)
linkTitle: Cluster state (new, existing)
weight: 1
---
EASEGRESS_STATE_FLAG:                  --state-flag

---
title: List of peer URLs of primary members. Define this only, when cluster-role is secondary.
linkTitle: List of peer URLs of primary members. Define this only, when cluster-role is secondary.
weight: 1
---
EASEGRESS_PRIMARY_LISTEN_PEER_URLS:    --primary-listen-peer-urls

---
title: Maximum size in bytes for cluster synchronization messages.
linkTitle: Maximum size in bytes for cluster synchronization messages.
weight: 1
---
EASEGRESS_MAX_CALL_SEND_MSG_SIZE:      --max-call-send-msg-size

---
title: Address([host]:port) to listen on for administration traffic.
linkTitle: Address([host]:port) to listen on for administration traffic.
weight: 1
---
EASEGRESS_API_ADDR:                    --api-addr

---
title: Flag to use secure transport protocol(https).
linkTitle: Flag to use secure transport protocol(https).
weight: 1
---
EASEGRESS_TLS:                         --tls

---
title: Flag to set the certificate file for https.
linkTitle: Flag to set the certificate file for https.
weight: 1
---
EASEGRESS_CERT_FILE:                   --cert-file

---
title: Flag to set the private key file for https.
linkTitle: Flag to set the private key file for https.
weight: 1
---
EASEGRESS_KEY_FILE:                    --key-file

---
title: Flag to set lowest log level from INFO downgrade DEBUG.
linkTitle: Flag to set lowest log level from INFO downgrade DEBUG.
weight: 1
---
EASEGRESS_DEBUG:                       --debug

---
title: Flag to set whether to disable access logs
linkTitle: Flag to set whether to disable access logs
weight: 1
---
EASEGRESS_DISABLE_ACCESS:              --disable-access

---
title: The time interval to dump running objects config, for example: 30m
linkTitle: The time interval to dump running objects config, for example: 30m
weight: 1
---
EASEGRESS_OBJECTS_DUMP_INTERVAL:       --objects-dump-interval

---
title: List of configuration files for initial objects, these objects will be created at startup if not already exist.
linkTitle: List of configuration files for initial objects, these objects will be created at startup if not already exist.
weight: 1
---
EASEGRESS_INITIAL_OBJECT_CONFIG_FILES: --initial-object-config-files

---
title: Path to the home directory.
linkTitle: Path to the home directory.
weight: 1
---
EASEGRESS_HOME_DIR:   --home-dir

---
title: Path to the data directory.
linkTitle: Path to the data directory.
weight: 1
---
EASEGRESS_DATA_DIR:   --data-dir

---
title: Path to the WAL directory.
linkTitle: Path to the WAL directory.
weight: 1
---
EASEGRESS_WAL_DIR:    --wal-dir

---
title: Path to the log directory.
linkTitle: Path to the log directory.
weight: 1
---
EASEGRESS_LOG_DIR:    --log-dir

---
title: Path to the member directory.
linkTitle: Path to the member directory.
weight: 1
---
EASEGRESS_MEMBER_DIR: --member-dir

---
title: Path to the CPU profile file.
linkTitle: Path to the CPU profile file.
weight: 1
---
EASEGRESS_CPU_PROFILE_FILE:             --cpu-profile-file

---
title: Path to the memory profile file.
linkTitle: Path to the memory profile file.
weight: 1
---
EASEGRESS_MEMORY_PROFILE_FILE:          --memory-profile-file

---
title: Number of object statuses to update at maximum in one transaction.
linkTitle: Number of object statuses to update at maximum in one transaction.
weight: 1
---
EASEGRESS_STATUS_UPDATE_MAX_BATCH_SIZE: --status-update-max-batch-size

Configuration tips (optional)

What is a good size for the cluster?

It is a good practice to choose an odd number (1,3,5,7,9) of primary nodes, to tolerate failures of primary nodes. This way the cluster can stay in a healthy state, even if the network partitions. With an even number of primary nodes, the cluster can be divided into two groups of equal size due to network partition. Then neither of the sub-clusters have the majority required for consensus. However, with the odd number of primary nodes, the cluster cannot be divided into two groups of equal size and this problem cannot occur.

For the secondary nodes, there are no constraints for the number of nodes. Secondary nodes do not participate consensus vote of the cluster, so their failure does not affect the cluster health. Adding more (secondary) nodes does still increase the communication between nodes.

Can a number of primary members scale up?

Please note that it is not recommended to add additional node with primary cluster role, but primary nodes should be started at cluster start up. When scaling up the cluster, it is recommended to add and remove secondary cluster members.

What are advertise-peer-urls and listen-client-urls?

advertise-client-urls, listen-client-urls, listen-peer-urls and initial-advertise-peer-urls are arguments necessary for primary cluster members to communicate with other peers (members).

argument	description
advertise-client-urls	client URLs member advertises to the rest of the cluster
listen-client-urls	URLs that member listens for client traffic
listen-peer-urls	URLs that member listens for peer (other primary members) traffic
initial-advertise-peer-urls	peer (other primary members) URLs member advertises to the rest of the cluster

These arguments are used for etcd server and client configuration. You can read more about them in etcd documentation.

Why primary member needs more arguments than secondary member ?

Here’s a drawing that illustrates the difference.

Primary members need to synchronize with peers (other primary members). Meanwhile secondary can read and update the state through one primary member.

How are etcd server and Easegress primary member related?

Easegress uses etcd distributed key-value store to synchronize the cluster state. The primary and secondary cluster roles have the following relation with etcd:

Easegress cluster role	primary	secondary
etcd term	server	client

Primary member uses etcd server for cluster communication, while secondary member uses etcd client for this.