While front end development tends to get a lot of attention, we’d like to shed some light on the developers and engineers who work behind the curtain to ensure that the website or application you’re seeing is firing on all cylinders.

Back end developers play a critical role on Kenzan teams, managing and building everything from servers to databases and all the things in between. To learn a little bit about what it’s like to be a Back End Developer, I spoke to one Kenzanite, Nicholas Sledgianowski (aka Sledge), who’s been at Kenzan for 3 years and works out of our Denver office.

If you’re interested in joining our team of back end developers and are in the Denver area, RSVP for our Back End Developer Job Fair on December 13.  For those that aren’t in Denver, please visit our careers page for more information on our open positions.

Tell us a little bit about yourself and your experience.
I have worked as a software engineer for around 5 years with exposure to both the front and back end of websites. My original plan was to be a physics researcher but I switched to software during my college years. I live in Denver and enjoy hiking and ballroom dance.

How did you get into computer science/development?
In high school I tried to get into game programming with C++, I did not get very far but I took an introductory C++ class in college that got me hooked on programming.

What kind of projects do back end developers work on at Kenzan?
Typically, our setup is a single page web application with many microservice APIs behind it. Back end developers mainly work on Java APIs and sometimes data pipelines supporting microservices. We would be responsible for supporting logins, authentication, designing database tables, database migrations, debugging production issues and caching systems.  At Kenzan, back end developers can expect to be involved in everything behind the JavaScript running on the client’s machine.

What’s been your favorite project? Why?
My favorite project was a data pipeline we built to provide a unified user data layer on top of 3 legacy user databases. The client had undergone a merger and was moving all users onto a unified site. We had a Kafka topic which the legacy systems published changes to, then an Akka cluster handled denormalizing the data and writing it to a Cassandra database. I got to spend some time optimizing our Cassandra driver’s write throughput which was a blast!

What’s your favorite programming language? Why?
My favorite programming language is Clojure. It is really beautiful to work with and has very powerful abstraction and polymorphism features. Clojure is not the best language for web development but it is great in other contexts. I recommend checking it out whether or not you have played with Lisp before.

What technologies are in your tool box?
It is more of a question of what technologies are not in my tool box. I have done a little front end, a little mobile development, a lot of back end development and DevOps. I try to stay away from the front end and the JavaScript ecosystem, it moves quickly and I only have so much time. I like to know everything about everything so it is important for me to ignore things to avoid being stretched too thin.

What’s a new technology that you’re psyched about?
I am pretty excited in GRPC and its multi-language compilation capabilities. Other than that I want to try and build things with all the machine learning APIs we have available nowadays.

What qualities do you think make you a good back end developer?
I am really good at reading which helps because there is a lot of code in a typical project and if you really want to know what is going on you need to read a lot of it. Code is written by people and you can get an idea of what patterns and thought processes went into a program from reading it.

Where did you learn your skills?
Most of my skills came from programming and the free resources on the internet. There are blog posts for everything you can imagine in information technology today. I am always reading about projects people have done online and trying out their open source software. A few good places to look for ideas are news.ycombinator.com, http://highscalability.com, and https://blog.acolyer.org/

What’s the best piece of advice you’ve received at Kenzan?
I think the best advice I have gotten is to use the dependency management tools in Maven and Gradle (ex. mvn dependency:tree) to help you figure out tricky dependency conflicts. Without that you can get stuck for days trying to work dependencies out. My best piece of advice is to read the book The Phoenix Project if you want to understand project managers, it is like fables for project managers.

To learn about Back End Developer opportunities at Kenzan, join us on December 13 for a job fair. RSVP here

To see other open positions at Kenzan, visit our careers page.

Continuous delivery has quickly become the choice method for faster, safer and more frequent software deployments. As more and more tools come into play in this arena, developers are looking for new ways to enable this kind of software delivery and maximize its benefits. This guide will show how I created a bare-metal continuous-delivery appliance using Spinnaker, running on a Kubernetes cluster of “mini pcs”.

Why build a bare-metal cluster?

Because it’s fun! Many times when using a cloud platform, much of the magic gets abstracted behind dashboards and APIs. When you set up your own cluster from scratch, it really helps to connect the dots and learn about how the pieces fit together.

The cloud is not cheap. Running Spinnaker in the cloud is quite pricey due to the resource requirements. When we run this on our own hardware, we pay up front around as much as one month in the cloud, but we can run it forever!

Free up resources. There are tools like minkube that allow you to set up your own single node Kubernetes cluster on your laptop. However this ties up resources. It’s very nice being able to use an always running cluster on your network without needing to constantly “spin up and spin down” environments.

Total control. Running a cluster via minikube or GKE is very convenient. However with bare-metal we can tweak our setup to our heart’s content. Want to install an nfs server for persistent volumes on a node? Go for it! Want to experiment with the Ubuntu Kubernetes distribution? Install the iso on a node! There is less magic and more “nitty gritty”. It really helps you understand how things work from the core.

Put your spare compute to work. It’s really satisfying to have your own “on prem” equipment. How many of us have raspberry pis or old desktops/laptops just laying around? Instead of just collecting dust, we can add these nodes to our fleet. We can then run jobs or applications that distribute this load. You can put the cluster behind your router and host web sites or run home automation applications. All without the overhead of the cloud. Plus it looks really cool sitting on your desk!

What is Spinnaker?

http://www.spinnaker.io/ is set of microservices that make it easy to build continuous delivery pipelines. Contributors to the project include Netflix, Google, Microsoft and Kenzan.

The project brings together best practices and patterns for easily deploying immutable infrastructure style software. Deploy targets can be instances or containers running on a multitude of platforms including AWS, GCP, Azure and Kubernetes.

Why Kubernetes?

Spinnaker can be run from any of the above platforms, however due to the nature of the resources needed it can be quite expensive. Kubernetes allows us to set up our own “cloud” on bare metal. We can then use our Spinnaker instance to easily deploy to other cloud platforms or clusters. It’s also pretty neat having a self contained “appliance” running Spinnaker. Hardware prices are constantly falling and it is pretty fun experimenting with software on our own “datacenter”.

Choosing hardware

While I was able to get the cluster turned up with the first version of this guide, The “Stick pcs” proved to be too weak on the memory requirement. I needed nodes that had at least 4gb memory. After some searching I decided on three “nexbox” pcs.

https://www.aliexpress.com/store/product/1Set-Nexbox-T9-Smart-TV-Box-Z8300-1-84GHz-4-Cores-Win-10-Mini-PC-4GB/2130214_32658221265.html

• 4GB memory
• Quad core atom processor
• 64GB SSD
• Ethernet port

It took a while for the boxes to arrive from Aliexpress but I was excited to get started when they did.

Installing Ubuntu

Unlike the tv-sticks, these boxes came with windows installed. That was not good for the cluster, so I began by trying to install Ubuntu server. Unfortunately since the chipset in these machines was “cherry trail” it had limited Linux compatibility. The nic did not work at all during install.

Thankfully, after some searching I was able to find an Ubuntu image with a modified kernel to support the chipset: http://linuxiumcomau.blogspot.com/search?updated-min=2017-01-01T00:00:00%2B11:00&updated-max=2018-01-01T00:00:00%2B11:00&max-results=2

Burning the image to a thumb drive, I was then able to hold the ESC key and boot from the drive to install.

With Ubuntu installed on the boxes I was now able to install docker on each node.

Installing Kubernetes

apt-get update
apt-get install docker.io

I leveraged the docker-multinode scripts to get Kubernetes installed along with heapster and the dashboard.

https://github.com/kubernetes/kube-deploy

Installation is relativly simple, you run the master.sh script on one node and worker.sh on the remaining two nodes.

Tunneling 8080 into the master node and the dashboard displayed like a charm.

Installing Spinnaker

I then was able to make some minor modifications to the “spinikube” specs and get Spinnaker installed.

https://github.com/kenzanlabs/spinikube

Overall, I’m very happy with how the cluster turned out. It’s great to be able to have a dedicated cluster without wasting resources on vms. I’m looking forward to running more workloads and monitoring performance. The next step will be to experiment with getting persistent volume storage in place with ceph or gluster. It will be great to take advantage of all the storage on the nodes. Stay tuned for part 2 of this guide, where we leverage Spinnaker to do a deploy on the cluster, along with some other advanced functionality.

Chad Moon is a platform engineer at Kenzan, based out of the Denver office. Specialties include crafting continuous delivery pipelines and containerizing all the things. Current work includes integrating Jenkins, Spinnaker and Kubernetes for large enterprise clients.

Have questions about building your own Spinnaker-Kubernetes cluster, or just about Continuous Delivery in general? Comment here or tweet at us: @kenzanmedia.

For more information about Kenzan services, contact info@kenzan.com

As more businesses prepare to make a digital transformation, containers have become the choice cloud computing architecture for faster, more portable and reliable deployments. With the growing interest in containerization, the question arises about how containers integrate with existing infrastructure. In this post, we will look at how containerization affects service discovery and present a network routing solution that allows NetflixOSS Eureka to provide unified discovery between both containers and with VM-based services.

Kenzan specializes in cloud technologies with extensive experience in Amazon Web Services (AWS). We have adopted a number of tools from the NetflixOSS stack for use in AWS, such as Zuul, Ribbon, and Eureka. The discovery service feature of Eureka allows us to build dynamically scalable AWS environments without the need to setup fixed routing and load balancing infrastructure.

Docker introduced a new networking layer that changes everything we know about networking in the cloud. This makes discovery and routing with Eureka challenging. Containers have their own IP addresses, belong on a different subnet, and are only routable from the host running the Docker daemon.

We have experimented with tools like flanneld that create virtual networking layers between Docker hosts, allowing for cross-host communication between Docker containers. Flanneld is easy to setup and does as advertised, but requires hosts wishing to network with containers to be running the flannel daemon. All-in-one Docker solutions like Kubernetes do everything from networking containers to orchestrating and managing multiple Docker hosts, but still cannot network with containers from outside the Kubernetes cluster.

What we are looking for is to spin up containers like we do with EC2 instances, have them register with a discovery service, and allow us to send traffic from anywhere in the VPC. Let’s start simple with a cluster of Docker hosts, which is something that EC2 Container Service (ECS) will provide us. We launch a few applications as Docker instances using ECS, but we get containers that can’t talk to each other and can’t be reached from any external service.

As an application in a discovery-based world, you need to tell the discovery service who you are and how others can reach you. This is easy on EC2 instances because the application can provide the host IP address and the port it’s listening on. Containers in Docker are given IP addresses that are only routable to containers running on the same Docker daemon. We can expose internal container ports as host ports, but that port (which port? That = ?)has to be static and non-conflicting with other containers on the same host. e want to be dynamic and not have to remember which ports are used versus which ports are free.

ECS provides an option to expose containers through an Elastic Load Balancer (ELB). Issues with ELB’s include consuming several VPC IP addresses, requiring management of limits on how many ELB’s can be created, and adding additional AWS costs. Our applications now have to remember a series of hostnames representing ELB endpoints for each environment, adding more configuration overhead. ELB’s  have the advantage of security groups, which is something we may expect to lose in a container world. The new networking layer on top of Docker does not play nicely with network based firewalls like security groups in AWS.

The goal is to find a non-conflicting dynamic way for containers deployed to a cluster of Docker hosts to identify themselves to a discovery service and have their identity be reachable. To achieve this, we need a tool that will find other containers on the host and route traffic based on a series of rules. Traefik is that tool. Traefik is a discovery based HTTP reverse proxy and load balancer that can discover Docker containers with minimal configuration, as well as several other means of discovery.

Let’s look at what it takes to get Traefik running with a Docker based backend:

$ docker run -d -p 80:80 -p 8080:8080 -v /var/run/docker.sock:/var/run/docker.sock traefik –web –docker –docker.domain=docker.localhost

That’s it.  The front end is now listening on port 80. Notice what we did here is we mounted the docker.sock file as a volume launching the container. This gives Traefik API access to the Docker daemon so that it can find other containers.

Alright, now it is time to add some containers for Traefik to find:

$ docker run -d –name nginx -l traefik.port=80 nginx

We added an Nginx container, and added a Label of traefik.port=80. Traefik will use the Docker metadata API exposed through the mounted unix socket to find container labels and bind the listener to port 80 that Nginx is listening on.

The Nginx container can be seen on the Traefik admin page listening on port 8080

Notice how the Rule is Host:nginx.docker.localhost. This is a combination of the container name we provided to Nginx with the –name argument. The docker.localhost part of the domain came from the –docker.domain=docker.localhost we gave to Traefik at startup.

Running a curl to the Docker host with a Host header of nginx.docker.localhost returns the Nginx welcome page.

$ curl -H “Host: nginx.docker.localhost” 192.168.33.10
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
….

Traefik is routing requests with this Host header to our Nginx container. We didn’t tell Traefik to do that, but it did (and that’s alright).

We can use other labels attached to the Nginx container to control how Traefik routes requests to it. Let’s try a path based rule.

$ docker run –name nginx -l traefik.port=80 -l traefik.frontend.rule=PathPrefix:/nginx nginx

Now all requests with the path prefix of /nginx will go to our Nginx container

$ curl 192.168.33.10/nginx/test
<html>
<head><title>404 Not Found</title></head>
….

We of course get a 404 since our nginx doesn’t have a /test resource, but we can see in the Nginx logs that the requests are coming through.

172.17.0.2 – – [10/Jan/2017:22:48:37 +0000] “GET /nginx/test HTTP/1.1” 404 169 “-” “curl/7.49.1” “192.168.33.1”

That was a quick proof-of-concept to show how quickly we got Traefik running as a discovery based routing service with Docker. By putting a Traefik container on every Docker host, we can  dynamically setup routes to our containers running on those hosts. All we need to do now is identify our containers to the discovery service as the host IP address and the port that Traefik is listening on. The applications calling the services have to remember to include the path prefix in the request. Below is an architecture diagram showing Traefik set up on multiple Docker hosts.

Darren Bathgate is a technical architect at Kenzan. Over the course of his 5+ years at Kenzan, Darren has worked extensively with Java, MySQL, PHP, Cassandra, Node.js, oracle, Jenkins, Netflix OSS and Docker.