Playing with ETCD cluster in Docker on Local

I’ve started to write some management component last week. We would like to utilize CoreOs with the whole stack, as much as possible, at least within such early phase of our project.

The core component of our solution is ETCD – distributed key value store. Something like my favorite piece of software – Redis. Word ‘distributed’ means that the core of all things within your solution needs to be synchronized or ‘consensused’. ETCD uses Raft. I’d love to know how my desired component works in real environment where everything can die.

In the age of docker – where every piece of software is docker-ized, it’s pretty simple to start ETCD cluster on local in a second. Following piece of code starts three etcd instances linked together in one cluster.

docker run -d -p 4001:4001 -p 2380:2380 -p 2379:2379 –net=host –name etcd0 quay.io/coreos/etcd:v2.0.3 \
 -name etcd0 \
 -advertise-client-urls http://localhost:2379,http://localhost:4001 \
 -listen-client-urls http://localhost:2379,http://localhost:4001 \
 -initial-advertise-peer-urls http://localhost:2380 \
 -listen-peer-urls http://localhost:2380 \
 -initial-cluster-token etcd-cluster-1 \
 -initial-cluster etcd0=http://localhost:2380,etcd1=http://localhost:2480,etcd2=http://localhost:2580

docker run -d -p 4101:4101 -p 2480:2480 -p 2479:2479 –net=host –name etcd1 quay.io/coreos/etcd:v2.0.3 \
 -name etcd1 \
 -advertise-client-urls http://localhost:2479,http://localhost:4101 \
 -listen-client-urls http://localhost:2479,http://localhost:4101 \
 -initial-advertise-peer-urls http://localhost:2480 \
 -listen-peer-urls http://localhost:2480 \
 -initial-cluster-token etcd-cluster-1 \
 -initial-cluster etcd0=http://localhost:2380,etcd1=http://localhost:2480,etcd2=http://localhost:2580

docker run -d -p 4201:4201 -p 2580:2580 -p 2579:2579 –net=host –name etcd2 quay.io/coreos/etcd:v2.0.3 \
 -name etcd2 \
 -advertise-client-urls http://localhost:2579,http://localhost:4201 \
 -listen-client-urls http://localhost:2579,http://localhost:4201 \
 -initial-advertise-peer-urls http://localhost:2580 \
 -listen-peer-urls http://localhost:2580 \
 -initial-cluster-token etcd-cluster-1 \
 -initial-cluster etcd0=http://localhost:2380,etcd1=http://localhost:2480,etcd2=http://localhost:2580

The inspiration is obvious, but this stuff simply runs everything on your computer.  Parameter –net=host provides full transparency from port&network point of view.

You can now use following URL in a browser:

http://localhost:4101/v2/keys/?recursive=true

Good thing is also to check all members of your cluster. You will kill them later.

http://localhost:2379/v2/members

You can easily delete all keys in XYZ namespace using curl once you did you tests. Note that you can delete only one of your keys so you can’t perform following command on your root namespace.

curl http://127.0.0.1:2379/v2/keys/XYZ?recursive=true -XDELETE

I also prefer to see http status code as ETCD uses http status codes.

curl -v http://127.0.0.1:2379/v2/keys/XYZ

In advance to status codes, it always returns a json with their own errors codes. See a snippet at the end of the following listing. You can get something similar to:

* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1…
* Connected to localhost (127.0.0.1) port 2379 (#0)
> GET /v2/keys/XYZ HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:2379
> Accept: */*

< HTTP/1.1 404 Not Found
< Content-Type: application/json
< X-Etcd-Cluster-Id: 65a1e86cb62588c5
< X-Etcd-Index: 6
< Date: Sun, 01 Mar 2015 22:55:14 GMT
< Content-Length: 69

{“errorCode”:100,”message”:”Key not found”,”cause”:”/XYZ”,”index”:6}
* Connection #0 to host localhost left intact

At the end of playing with ETCD cluster, you will probably want to remove all etcd’s containers. I use simple script which removes every docker container, but you can improve it using grep to remove only those hosting ETCD.

sudo docker rm -f `docker ps –no-trunc -aq`

The last interesting thing is the performance. I’ve reminded Redis which can handle one million of transactions per second using one thread. I was surprised when ETCD responded usually in 20-30ms. Much worse fact is that I’ve also encountered client timeouts because of 400-500ms RT per request. Raft is obviously not for free. But the purpose of ETCD is massive reading scalability. Well, good to know.

2 thoughts on “Playing with ETCD cluster in Docker on Local

  1. Hello Martin, I have to implement an ETCD (V3) cluster on my openstack project, every VMs are reachable from each other, (ping and ssh).Then, a few weeks ago, I followed the Coreos docs in order to start a cluster in containers on a static way, with a bash file for the environment vars.So here is my conf file:#!/bin/bashexport ETCD_VERSION=v3.0.0export TOKEN=http://192.168.0.5/753a8bf0-0ba5-43ac-b5ac-a2a47e430c11export CLUSTER_STATE=newexport NAME_1=etcd-server1export NAME_2=etcd-server2export NAME_3=etcd-server3export HOST_1=192.168.0.5export HOST_2=192.168.0.6export HOST_3=192.168.0.7export CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380As you can see, I wanted to use one of my tecd nodes to be the discovery service with a token, in my case I uses the command UUID=$(uuidgen).So as you can see, my etcd-server1 IS the one with the token to contact in order to know every members of the cluster.But when I run this command on each VM:docker run -dt –net=host –name etcd quay.io/coreos/etcd:V3.0.0 /usr/local/bin/etcd –data-dir=data.etcd –name ${THIS_NAME} –initial-advertise-peer-urls http://${THIS_IP}:2380 –listen-peer-urls http://${THIS_IP}:2380 –advertise-client-urls http://${THIS_IP}:2379 –listen-client-urls http://${THIS_IP}:2379 –initial-cluster ${CLUSTER} –initial-cluster-state ${CLUSTER_STATE} –initial-cluster-token ${TOKEN}My etcd-server1 has stopped (see this dockers logs):ubuntu@etcd-server1:~$ docker logs etcd2016-08-10 14:05:22.910217 I | etcdmain: etcd Version: 3.0.02016-08-10 14:05:22.910468 I | etcdmain: Git SHA: 6f48bda2016-08-10 14:05:22.910571 I | etcdmain: Go Version: go1.6.22016-08-10 14:05:22.910669 I | etcdmain: Go OS/Arch: linux/amd642016-08-10 14:05:22.910790 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 12016-08-10 14:05:22.911139 I | etcdmain: listening for peers on http://192.168.0.5:23802016-08-10 14:05:22.911315 I | etcdmain: listening for client requests on 192.168.0.5:23792016-08-10 14:05:22.918816 I | etcdmain: stopping listening for client requests on 192.168.0.5:23792016-08-10 14:05:22.918983 I | etcdmain: stopping listening for peers on http://192.168.0.5:23802016-08-10 14:05:22.919092 I | etcdmain: –initial-cluster must include etcd-server1=http://192.168.0.5:2380 given –initial-advertise-peer-urls=http://192.168.0.5:2380My question is this one: Do you think I should insert the token in a different manner? For example run the discovery container with “sh” command in order to insert the key? But I do not know how :/Thank you for your time.Benjamin

    Like

  2. Hello Martin, I have to implement an ETCD (V3) cluster on my openstack project, every VMs are reachable from each other, (ping and ssh).Then, a few weeks ago, I followed the Coreos docs in order to start a cluster in containers on a static way, with a bash file for the environment vars.So here is my conf file:#!/bin/bashexport ETCD_VERSION=v3.0.0export TOKEN=http://192.168.0.5/753a8bf0-0ba5-43ac-b5ac-a2a47e430c11export CLUSTER_STATE=newexport NAME_1=etcd-server1export NAME_2=etcd-server2export NAME_3=etcd-server3export HOST_1=192.168.0.5export HOST_2=192.168.0.6export HOST_3=192.168.0.7export CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380As you can see, I wanted to use one of my tecd nodes to be the discovery service with a token, in my case I uses the command UUID=$(uuidgen).So as you can see, my etcd-server1 IS the one with the token to contact in order to know every members of the cluster.But when I run this command on each VM:docker run -dt –net=host –name etcd quay.io/coreos/etcd:V3.0.0 /usr/local/bin/etcd –data-dir=data.etcd –name ${THIS_NAME} –initial-advertise-peer-urls http://${THIS_IP}:2380 –listen-peer-urls http://${THIS_IP}:2380 –advertise-client-urls http://${THIS_IP}:2379 –listen-client-urls http://${THIS_IP}:2379 –initial-cluster ${CLUSTER} –initial-cluster-state ${CLUSTER_STATE} –initial-cluster-token ${TOKEN}My etcd-server1 has stopped (see this dockers logs):ubuntu@etcd-server1:~$ docker logs etcd2016-08-10 14:05:22.910217 I | etcdmain: etcd Version: 3.0.02016-08-10 14:05:22.910468 I | etcdmain: Git SHA: 6f48bda2016-08-10 14:05:22.910571 I | etcdmain: Go Version: go1.6.22016-08-10 14:05:22.910669 I | etcdmain: Go OS/Arch: linux/amd642016-08-10 14:05:22.910790 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 12016-08-10 14:05:22.911139 I | etcdmain: listening for peers on http://192.168.0.5:23802016-08-10 14:05:22.911315 I | etcdmain: listening for client requests on 192.168.0.5:23792016-08-10 14:05:22.918816 I | etcdmain: stopping listening for client requests on 192.168.0.5:23792016-08-10 14:05:22.918983 I | etcdmain: stopping listening for peers on http://192.168.0.5:23802016-08-10 14:05:22.919092 I | etcdmain: –initial-cluster must include etcd-server1=http://192.168.0.5:2380 given –initial-advertise-peer-urls=http://192.168.0.5:2380My question is this one: Do you think I should insert the token in a different manner? For example run the discovery container with “sh” command in order to insert the key? But I do not know how :/Thank you for your time.Benjamin

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this:
search previous next tag category expand menu location phone mail time cart zoom edit close