Monthly Archives: November 2015

Building go-lang project /Kubernetes Ingress/ from scratch with no go experience

I have been working with Kubernetes and I wanted to build it’s contrib yesterday. However, nginx implementation of Kubernetes’ Ingress is written in go-lang. Even though I needed to change some const string, it required recompilation.

Go is not java, and go building system is not maven. Setting up the environment was not straightforward. I was facing couple troubles but I’m going to take it from the beginning. My laptop uses ubuntu 15.04 – well 15.10 since 9pm 🙂 – and I have never installed go lang yet.

Go lang installation on Ubuntu

First of all, you need to install go lang. You can use official repo, but it contains older version 1.3. However, do not install it using apt-get as kubernetes or it’s dependencies require higher version of go-lang. Of course, I originally installed version 1.3  but some fatal error occurred later. It forced me do to the manual installation anyway.

Here is simple tutorial.

https://gist.github.com/MartinPodval/948818a0635ba0759324.js
Last two lines affect only the current terminal. You should update /etc/environment in the case you consider to use go-lang in the future.

Then, you can try to get version of go via:

$ go version
go version go1.5.1 linux/amd64

Go-lang compilation and building

Now, you can pull your git project, e.g.:

git clone https://github.com/kubernetes/contrib


 There is a build command in go-lang. You can type it into you terminal:

https://gist.github.com/MartinPodval/998be55bf77f45b4703a.js
What’s wrong? We have not setup GOPATH yet. This directory is something like maven cache. It’s pretty simple, you can just point the variable to your home, e.g.

export GOPATH=/home/martin/.go

Try to build it again. There are no errors, but nothing is happening. I found in some stackoverflow discussion that there is a get command as well. Unfortunately, build is consist of get anyway. Then I found -v as verbose.

https://gist.github.com/MartinPodval/cbe92056b842356d1b35.js
Actually, we are on right track. The fetching of dependencies takes a time so it seemed that go build or go get did not work but that’s not true.

That’s all, now building should work:

https://gist.github.com/MartinPodval/d10c864185bc2ce91ffe.js

Go-lang 1.3 vs 1.5

I just wanted to discuss why to prefer manual installation over ubuntu repository. For the record. Building kubernetes project using go-lang 1.3 from ubuntu official repo failed:

https://gist.github.com/MartinPodval/6b60130ae64dfbfceb32.js

Using CoreOS stack and Kubernetes #2: Why use CoreOS as Cloud Operating System

I’d like to deal in this part with potential benefits resulting from using CoreOS as an operating system in your cloud deployment. You can install kubernetes on various operating systems so you can make a decision what to choose. So why CoreOS? What is my experience?

Etcd, Fleet and Flannel Preinstalled

First reason is obvious. CoreOS always provides latest version of all components in Kubernetes cluster. 
My experience: we have profited from pre-installed components from the beginning. E.g. in early stages when etcd was coming with new beautiful and powerful API (v.2), they put both – old and new – versions together so we just enabled one of them. The setup of all components together is not very simple so you can save couple hours by choosing preinstalled and pre-setuped CoreOS.

No Package Manager, Read Only Partitions

It sounds more like disadvantage than benefit, but …

Look at CoreOS releases what it consist of.

Fore example, CoreOS includes basic linux utils so you can employ many popular command line tools. But it’s not recommended to install anything else. Take what is installed and all machines within the cluster can be easily added, removed and/or replaced. All parts of your application are supposed to be distributed as docker containers.
CoreOS installation also use a concept of nine disk partitions. Some of them are read only, some of them contain operating system. This forces an administrator to keep mutable data on one of them. This, again, improves node replaceability.
My experience: this is great for operations. It’s matter of few seconds to add a new node. However, it’s sometimes tough to work with CoreOS when you are used to rely on some tools, like htop. Speaking of which, there is nothing against manual download anyway, e.g. via the cloud config.

Online Updates

There is a great update methodology. You can setup a CoreOS node to do an automatic update. What does it mean in real?
You choose an update channel (alpha, beta, stable) and CoreOS does automatic checking of new versions as well. You can manually use tool update_engine_client to manage updates from command line. This is useful for debugging in early stages when you did not setup updates properly and they might fail.
Once the update engine detects a new version, it immediately starts to download new bytes. There is a notion of active and passive partitions. The current boot runs from active partition, downloading uses passive one.
CoreOS needs a reboot to apply the new version of the operating system. However, consider running cluster of many and many nodes. What would happen when they downloaded new operating system version? They would reboot all together!
Here is locksmith tool. This stuff utilizes etcd persistent storage to do simple semaphore for all running and potentially rebooting CoreOs nodes. In short, this distributed lock guarantees that only one machine is being rebooted in a time.
My experience: this is one of best things on CoreOS. You are just subscribed on some channel with proper reboot strategy and your cluster is continually up-to-date. Either linux kernel, fleet or etcd, linux tool or newly added Kubelet.
We have also encountered problems with one of new versions of CoreOS. For examples, there was a new version of golang and docker started to hang once it finished an image pulling. You can manually rollback or downgrade CoreOS version back. This tutorial just switch current node to passive read-only disk partition with previous version of CoreOS.

Cloud Configuration File

It’s always pretty long procedure to setup and configure a machine when it’s just installed with fresh operating system. Therefore, CoreOS brings with concept of cloud config files.

The point is to have the only file which contains the whole configuration of a node.

I’ll dedicate one chapter to this concept. However, it’s usual to store following information in cloud configs:
  • setup CoreOS specifics, e.g. update channel, rebooting strategy etc.
  • adjust any systemd service
  • write files, like proxy setting, certificates etc.
  • setup node hostname
  • configure etcd, fleet, kubernetes or docker tools
My experience: it’s pretty useful to have one cloud config for the whole cluster. You can put it to some storage, your git repository or artifactory. All nodes can take this instance and apply the content during it’s boot. This guarantees that all nodes have same configuration. 
There is a lot of other useful things on CoreOS but these above were major. I’d like to dedicate next article to the installation.

Here is a link to the whole series.

Using CoreOS stack and Kubernetes #1: Introduction

We were lucky enough in December 2014 to join the group of teams who use CoreOS stack and Kubernetes on their way to become next generation of cloud infrastructure. It has been almost one year so I’d like to provide a article series about our experience with the whole stack.

The Motivation

You usually want to model your business domain, provide useful APIs, break your application into pieces, services, and so on. Well, it’s your work.

The distributed computing is one of most challenging disciplines in the computer science. Why is that? Because of an asynchronicity in the form of remote calls among distributed components. There are no locks like in your favorite languages. However, there are remote calls with no guarantees of any response or in any time.

It’s pretty challenging to provide high-available application, with no downtime during updates, crashes. The application which scales according to the needs. The application with guarantees any data consistency.

What are typical questions and considerations when you start to build such app?

  • how can I run exactly 3 instances of a service in my app? 
  • how can I detect that some instance failed? 
  • how can I run a new replica instead of dead one? 
  • what if there are more than 3 instances because the dead replica was not so much dead and it’s now back in the cluster? 
  • what if there are two replicas – dead one and new one – which process same part of the data? 
  • how can I guarantee that all replicas can see the same configuration? 
  • where can service B discover a link to running service A? 
  • where can service B discover new instance of service A because the first one failed? 
  • how can I install all that mess to one operating system? 

I could write many and many questions like these above. CoreOS and Kubernates allows you to address many of these questions.

CoreOS stack and Kubernates provide well tested but tiny platform for your cluster/cloud infrastructure. You can focus on your business not on the infrastructure.

Components

Here is a diagram how all tools fits together:

  • CoreOS :It’s just very simple linux distribution prepared for cluster/cloud deployment.
  • Fleet is responsible for running units (services) on remote nodes
  • Etcd is distributed key value store using Raft consensus. The purpose is to store the configuration in namespaces. I’ve already wrote some articles about etcd
  • Flannel allows to provide private networking among nodes – or docker container in this case
  • Kubernetes uses all tools together to provide cluster management. You can describe your application via kubernetes descriptors and use kubectl or REST API to run, scale or fail-over your app. Obviously in cooperation with the application. One can say that it’s PaaS for dockerized application. And (s)he would be right.

What should I read to become more familiar with all these?

If you would have only 30 minutes check out this video:

What’s next?

I’d like to write article series about our experience with CoreOS and Kubernetes. I’d like to deal with the installation in next article.

Here is a link to the whole series.