The last presentation is about dynamic management of Kubernetes.
Kubernetes allows to manage large clusters which are composed of docker containers. And where is large computation power there is large amount of data throughput. However, there is still a need to spread the data throughput so it can reach and utilize particular docker container(s). This is called load balancing. Kubernetes supports two basic forms of load balancing. Internal among docker containers themselves and external for clients which call you application from outside environment – just users.
Internal Load Balancing with Kubernetes
Usual approach during the modeling of an application in kubernetes is to provide domain models for pods, replications controllers and services. Look at the great documentation if you are not aware of all these principles.
For simplification, pod is a set of docker containers which are always located on the same node. Replication controller allows and guarantees scalability. The last but not least is the service which hides all instances of pods – created by replication controller – behind a facade.
The concept of services brings very powerful system of internal load balancing.
Look at the picture above. Service A uses Service B. Common pattern is to provide some service discovery so some service A can look up some instance of service B’ endpoint and use it for later communication. However, service B has it’s own lifecycle and can fail forever or service B can have multiple running instance. Thus, a collection of instances of service B is changing during the time.
You need some boilerplate code to handle the algorithm describe above. Ohh, wait. Kubernetes already contains similar system.
Let say, service B has 3 pods – running docker containers. Kubernetes allows you to introduce one instance of service B to hide these three pods. In advance, it gives you one virtual IP address. Service A than can use this virtual IP address and kubernetes proxying system does the routing to a specific pod. Cool isn’t it?
The routing supports sticky session or round robin algorithm. It depends. The last thing which is up to you is to have clever client for your service, e.g. implemented with cooperation of hystrix, so it does re-trying or handle timeouts correctly.
External Load Balancing using Kubernetes
This type of load balancing is about routing the traffic from source clients to target proper pods. You have multiple options here.
Buy AWS/GCE Load Balancer
The load balancing and the networking itself is difficult system. Sharing public IPs, protocols to resolve hostnames to IPs, to resolve IPs to mac addresses, HA of LB and so on. It makes sense to buy load balancer as a service if you deploy your app in some well known cloud provider.
Thus, the services support load balancing type. Kubernetes then uses cloud provider specific API to configure the load balancer.
This is the simplest way. But what are your other options if you are not so lucky and you can’t do that? You have to deploy your own load balancer. Let’s review existing load balancers which can be deployed on premise.
Kube-proxy as external load balancer
I’ve already described the mechanism how kubernetes supports internal load balancing. Yeah! Is it possible to use it for external load balancing? Yes but there are some limitations.
Kube-proxy does exactly what we want but it’s very simple tcp load balancer – L4. I’ve detected some failures during some performance testing from external clients. The proxy just sometimes refused the connection or did not know target address of particular host and so on. However, there is no way how to configure kube-proxy, e.g. to setup some timeouts. Or there is also no way – as it’s TCP load balancer – how to do re-tries on http layer.
These all are reasons why I recommended to have clever service client within the internal load balancing section at the beginning. So I would still recommend to use kube-proxy as an internal load balancer but use some solid external load balancer implementation for external load balancing.
New version 1.1 of the system introduced ingress feature. However, it’s still beta.
Ingress is a new version of kubernetes resource. It’s API which allows you to define certain rules where to send a traffic. For example you can re-route a traffic from path http://host/example to a service called example. It’s pretty simple.
Ingress itself is not load balancer. It’s just a domain model and a tool to describe the problem. Thus there must be a service which absorbs this model and which does the load balancing but Ingress just helps you as a abstraction. Kubernetes team already developed ha-proxy implementation for this load balancing service.
Service Load Balancer
There is a project which exist from earlier kubernetes times called service load balancer. It’s under development for a year. It’s probably predecessor of Ingress. The load balancer also uses two levels of abstraction and any existing load balancer can stand as an implementation, ha-proxy is the only current implementation.
Vulcand is a new load balancer. It’s technology is compatible as the base configuration is stored within etcd – this allows to utilize watch command. Note that vulcand is http-only load balancer, tcp belongs among requested features. It’s also not integrated with kubernetes but here is a simple tutorial.
In fact, I have finally ended with developing of my own load balancer component which is based on existing system. It’s pretty easy.
There is a kubernetes java client from fabric8, you can use it to watch the kubernetes API and it’s then up to you which existing load balancer you want to leverage and which kind of config file you will generate. However, as it’s simple there are already some projects which do the same:
Nginx is similar technology to HAProxy so it’s easy to develop a component to configure Nginx load balancer as well. Nginx plus seems to support directly kubernetes but it’s a service.
I have been working with Kubernetes and I wanted to build it’s contrib yesterday. However, nginx implementation of Kubernetes’ Ingress is written in go-lang. Even though I needed to change some const string, it required recompilation.
Go is not java, and go building system is not maven. Setting up the environment was not straightforward. I was facing couple troubles but I’m going to take it from the beginning. My laptop uses ubuntu 15.04 – well 15.10 since 9pm 🙂 – and I have never installed go lang yet.
Go lang installation on Ubuntu
First of all, you need to install go lang. You can use official repo, but it contains older version 1.3. However, do not install it using apt-get as kubernetes or it’s dependencies require higher version of go-lang. Of course, I originally installed version 1.3 but some fatal error occurred later. It forced me do to the manual installation anyway.
Here is simple tutorial.
Last two lines affect only the current terminal. You should update /etc/environment in the case you consider to use go-lang in the future.
Then, you can try to get version of go via:
$ go version
go version go1.5.1 linux/amd64
Go-lang compilation and building
Now, you can pull your git project, e.g.:
git clone https://github.com/kubernetes/contrib
There is a build command in go-lang. You can type it into you terminal:
What’s wrong? We have not setup GOPATH yet. This directory is something like maven cache. It’s pretty simple, you can just point the variable to your home, e.g.
Try to build it again. There are no errors, but nothing is happening. I found in some stackoverflow discussion that there is a get command as well. Unfortunately, build is consist of get anyway. Then I found -v as verbose.
Actually, we are on right track. The fetching of dependencies takes a time so it seemed that go build or go get did not work but that’s not true.
That’s all, now building should work:
Go-lang 1.3 vs 1.5
I just wanted to discuss why to prefer manual installation over ubuntu repository. For the record. Building kubernetes project using go-lang 1.3 from ubuntu official repo failed:
I’d like to deal in this part with potential benefits resulting from using CoreOS as an operating system in your cloud deployment. You can install kubernetes on various operating systems so you can make a decision what to choose. So why CoreOS? What is my experience?
Etcd, Fleet and Flannel Preinstalled
No Package Manager, Read Only Partitions
Look at CoreOS releases what it consist of.
Cloud Configuration File
The point is to have the only file which contains the whole configuration of a node.
- setup CoreOS specifics, e.g. update channel, rebooting strategy etc.
- adjust any systemd service
- write files, like proxy setting, certificates etc.
- setup node hostname
- configure etcd, fleet, kubernetes or docker tools
Here is a link to the whole series.
We were lucky enough in December 2014 to join the group of teams who use CoreOS stack and Kubernetes on their way to become next generation of cloud infrastructure. It has been almost one year so I’d like to provide a article series about our experience with the whole stack.
You usually want to model your business domain, provide useful APIs, break your application into pieces, services, and so on. Well, it’s your work.
The distributed computing is one of most challenging disciplines in the computer science. Why is that? Because of an asynchronicity in the form of remote calls among distributed components. There are no locks like in your favorite languages. However, there are remote calls with no guarantees of any response or in any time.
It’s pretty challenging to provide high-available application, with no downtime during updates, crashes. The application which scales according to the needs. The application with guarantees any data consistency.
What are typical questions and considerations when you start to build such app?
- how can I run exactly 3 instances of a service in my app?
- how can I detect that some instance failed?
- how can I run a new replica instead of dead one?
- what if there are more than 3 instances because the dead replica was not so much dead and it’s now back in the cluster?
- what if there are two replicas – dead one and new one – which process same part of the data?
- how can I guarantee that all replicas can see the same configuration?
- where can service B discover a link to running service A?
- where can service B discover new instance of service A because the first one failed?
- how can I install all that mess to one operating system?
I could write many and many questions like these above. CoreOS and Kubernates allows you to address many of these questions.
CoreOS stack and Kubernates provide well tested but tiny platform for your cluster/cloud infrastructure. You can focus on your business not on the infrastructure.
- CoreOS :It’s just very simple linux distribution prepared for cluster/cloud deployment.
- Fleet is responsible for running units (services) on remote nodes
- Etcd is distributed key value store using Raft consensus. The purpose is to store the configuration in namespaces. I’ve already wrote some articles about etcd
- Flannel allows to provide private networking among nodes – or docker container in this case
- Kubernetes uses all tools together to provide cluster management. You can describe your application via kubernetes descriptors and use kubectl or REST API to run, scale or fail-over your app. Obviously in cooperation with the application. One can say that it’s PaaS for dockerized application. And (s)he would be right.
What should I read to become more familiar with all these?
- look into the documentation for the CoreOS
- there are tons of topics in the latest doc, cluster setup, installation, cloud configs etc.
- fleet documentation
- etcd docs
- flannel does (private) networking
- here is nice description of all components together
- watch videos from Kubernetes launch
- try to understand and learn how to manage systemd
- journalctl can show you log messages for every unit (process) managed by systemd
I’d like to write article series about our experience with CoreOS and Kubernetes. I’d like to deal with the installation in next article.
Here is a link to the whole series.