Continuous delivery has quickly become the choice method for faster, safer and more frequent software deployments. As more and more tools come into play in this arena, developers are looking for new ways to enable this kind of software delivery and maximize its benefits. This guide will show how I created a bare-metal continuous-delivery appliance using Spinnaker, running on a Kubernetes cluster of “mini pcs”.
Why build a bare-metal cluster?
Because it’s fun! Many times when using a cloud platform, much of the magic gets abstracted behind dashboards and APIs. When you set up your own cluster from scratch, it really helps to connect the dots and learn about how the pieces fit together.
The cloud is not cheap. Running Spinnaker in the cloud is quite pricey due to the resource requirements. When we run this on our own hardware, we pay up front around as much as one month in the cloud, but we can run it forever!
Free up resources. There are tools like minkube that allow you to set up your own single node Kubernetes cluster on your laptop. However this ties up resources. It’s very nice being able to use an always running cluster on your network without needing to constantly “spin up and spin down” environments.
Total control. Running a cluster via minikube or GKE is very convenient. However with bare-metal we can tweak our setup to our heart’s content. Want to install an nfs server for persistent volumes on a node? Go for it! Want to experiment with the Ubuntu Kubernetes distribution? Install the iso on a node! There is less magic and more “nitty gritty”. It really helps you understand how things work from the core.
Put your spare compute to work. It’s really satisfying to have your own “on prem” equipment. How many of us have raspberry pis or old desktops/laptops just laying around? Instead of just collecting dust, we can add these nodes to our fleet. We can then run jobs or applications that distribute this load. You can put the cluster behind your router and host web sites or run home automation applications. All without the overhead of the cloud. Plus it looks really cool sitting on your desk!
What is Spinnaker?
http://www.spinnaker.io/ is set of microservices that make it easy to build continuous delivery pipelines. Contributors to the project include Netflix, Google, Microsoft and Kenzan.
The project brings together best practices and patterns for easily deploying immutable infrastructure style software. Deploy targets can be instances or containers running on a multitude of platforms including AWS, GCP, Azure and Kubernetes.
Why Kubernetes?
Spinnaker can be run from any of the above platforms, however due to the nature of the resources needed it can be quite expensive. Kubernetes allows us to set up our own “cloud” on bare metal. We can then use our Spinnaker instance to easily deploy to other cloud platforms or clusters. It’s also pretty neat having a self contained “appliance” running Spinnaker. Hardware prices are constantly falling and it is pretty fun experimenting with software on our own “datacenter”.
Choosing hardware
While I was able to get the cluster turned up with the first version of this guide, The “Stick pcs” proved to be too weak on the memory requirement. I needed nodes that had at least 4gb memory. After some searching I decided on three “nexbox” pcs.
- 4GB memory
- Quad core atom processor
- 64GB SSD
- Ethernet port
It took a while for the boxes to arrive from Aliexpress but I was excited to get started when they did.
Installing Ubuntu
Unlike the tv-sticks, these boxes came with windows installed. That was not good for the cluster, so I began by trying to install Ubuntu server. Unfortunately since the chipset in these machines was “cherry trail” it had limited Linux compatibility. The nic did not work at all during install.
Thankfully, after some searching I was able to find an Ubuntu image with a modified kernel to support the chipset: http://linuxiumcomau.blogspot.com/search?updated-min=2017-01-01T00:00:00%2B11:00&updated-max=2018-01-01T00:00:00%2B11:00&max-results=2
Burning the image to a thumb drive, I was then able to hold the ESC key and boot from the drive to install.
With Ubuntu installed on the boxes I was now able to install docker on each node.
Installing Kubernetes
apt-get update
apt-get install docker.io
I leveraged the docker-multinode scripts to get Kubernetes installed along with heapster and the dashboard.
https://github.com/kubernetes/kube-deploy
Installation is relativly simple, you run the master.sh script on one node and worker.sh on the remaining two nodes.
Tunneling 8080 into the master node and the dashboard displayed like a charm.
Installing Spinnaker
I then was able to make some minor modifications to the “spinikube” specs and get Spinnaker installed.
https://github.com/kenzanlabs/spinikube
Overall, I’m very happy with how the cluster turned out. It’s great to be able to have a dedicated cluster without wasting resources on vms. I’m looking forward to running more workloads and monitoring performance. The next step will be to experiment with getting persistent volume storage in place with ceph or gluster. It will be great to take advantage of all the storage on the nodes. Stay tuned for part 2 of this guide, where we leverage Spinnaker to do a deploy on the cluster, along with some other advanced functionality.
Chad Moon is a platform engineer at Kenzan, based out of the Denver office. Specialties include crafting continuous delivery pipelines and containerizing all the things. Current work includes integrating Jenkins, Spinnaker and Kubernetes for large enterprise clients.
Have questions about building your own Spinnaker-Kubernetes cluster, or just about Continuous Delivery in general? Comment here or tweet at us: @kenzanmedia.
For more information about Kenzan services, contact info@kenzan.com