tech.harrywinser.com

A Social Network for Microservices

Quick question; how many places do you have to look to get a FULL view of your microservice? Git for the source code / README? Confluence for business documents? What about Jenkins for current build state? When was it last deployed? What's its current status in prod?

I did a count, and got around 6 different places to view. That's kinda a lot. For a single service.

Over the last few months I've been to be a fair number of devops style conferences, such as Pipeline Conf and JAX Devops, where people share their stories around Continuous Delivery and Integration. This has been awesome, and I've learned so much from so many people.

I was listening to talk about how X company made it from large big bang releases to continuous delivery, and how they had documentation for Operations to understand how to deal with their Microservices outside of normal work hours, in case something went wrong. This got me thinking of the question at the start, and also made me think; "if I had to give my microservice to someone totally new, where would they find info such as which services relied on it, and when was it last release". If only there was some format that everyone just... gets.

Then it hit me; Facebook.

Ok, hear me out on this. Pretty much everyone has Facebook, or has at least used it (I know that's quite a big assumption, but let's roll with it). It's got a UI that everyone knows roughly how to use. When you browse a profile, we EXPECT there to be photos, posts, and a bio. All of this maps rather well to microservices.

Think about it; a microservice has a README (a bio), it has consuming and/or providing services (friends), it has business documents (posts/blogs), it has things happening in its own little world such as deployments and builds and events (its wall), and some have service discovery (check-ins!). Whenever a service does something, it should be post to its own wall. It builds – that's a post. It deploys – that's a post. It dies in production – that's a post, and tags services that consume from it.

Suddenly, you've got this entire interaction network happening. And a pretty great view into what is going on within your whole platform, in an understandable way.

Last week, we had a hackathon at work, and I gathered a few devs together to see if we could create a proof of concept. Amazingly it worked. It made a whole lot of sense. It wasn't feature complete however, but suddenly we had this hub where people could see the status of their build, their last deployment (and by who), who last committed and when. We also managed to import the README's for each project, and a "friend" box which contained a services consuming and providing applications. We got some pretty great feedback, most around how easy it was to see what was going on with different applications.

I guess, what I'm aiming for is something that people enjoy using, while also creating a standardised UI that anyone can use. I think it's about bring documentation and metrics into a format that isn't all funky graphs, or nested trees hidden on some website. But instead a consistent format that anyone can access and use.

Anyway, that was my random idea. It's something I'm quite excited about, and if you'd like to know more please contact me!

Docker Secrets and Spring boot

Over the last few days, I've been doing a whole lot of work with Docker Swarm, AWS, and Spring boot. To give a little bit of context, I ended up using the Standard AWS docker template, and a Dynamo Db instance to host the data. Of course, this lead me to the age-old problem; how do I store my secrets securely and NOT within my app. And so, I thought I'd give Docker Secrets a go!

What are secrets?

I guess the first thing I should answer is the most obvious; what are secrets? Well, a Secret is a passcode, a password, or a key of some sort that is needed to access a data store or some form of precious item. It's a super bad idea to store this within your application (just search Github), and so some form of solution is required to load your secrets in at run time. A load of stuff has been written about Secrets, and what you should use and do and blah blah bah. In this blog however, I'm going take a look at my experience with Dockers home grown solution; Docker Secrets and using it with Spring boot.

Docker Secrets

You can find out a whole lot about Docker Secrets from their official website, but the brief overview is this: you create a secret, which is a key-value affair, from either a file or some form of input stream, and this is stored securely within the master node(s) of your Docker Swarm. From here, you can give a Secret to a Docker Service, where in the Containers for the Service, a new root folder will be added called "/run/secrets", where your Secrets will be loaded in file format. These files are your Secrets, where the name is the secret name, and the contents of the file is the actual Secret value. Simple enough, and means pretty much every programming language can access them.

My setup

So, back to my own experience. Overall, it was super simple to set up and get working. I have small Springboot application which needs to connect to a Dynamo Db instance. This means on startup of the application I need the accesskey and passkey to get into the database. I'm not going to store this in application, nor am I going to put it in a Docker Container / Image, as these will be hosted externally, and could be gotten hold of by some dastardly fiend.

So, using the @PostConstract annotation, I wrote a little method that looked for the default Docker Secrets folder, and loaded all the files found there into a HashMap, where the file names the keys, and the contents the values. From here, I could take the Hashmap and store them as properties in the application for use during other configuration steps. See, simple! Once I was confident the code worked, I went ahead and extracted it into a small micro-library which wraps all this up. It's not very feature complete, so pull requests welcome!

Things to take away

First off, I thought this was going to a much bigger pain to apply Secrets to my project. I'm confident that the Secrets are safe, and I know that as I add more masters and nodes I don't have to re-add the Secrets to each box; they should just propagate around the network. Also, if I need to update a secret, I know I can without having to go to each environment to update them. It just makes things super simple.

If you've got any feedback or suggestions, contact me in the usual ways!

Cheers,
Harry

What I learned from PIPELINECONF 2016

On the 26th of March, (that's a Wednesday) i attended PIPELINE CONF 2016. So i thought i'd go ahead and write a little blog about all the things i learned from the event.

First, i should probably outline what PIPELINE CONF is. Actually, it's probably best if i just quote the website;

'PIPELINE is an annual not-for-profit one day conference focussed on Continuous Delivery, held each spring at etc.venues Victoria in London.'

So, it's a group of people getting together to discuss and share their experiences with Continuous Delivery. A rather relevant topic considering how much of a "buzz word" it's been over the last few years.

So, without further ado, lets crack on with what I learned. Just a quick note, this isn't an any real order, and it's purely what i learned. If I've missed something, please get in touch. If I'm telling you to suck eggs... well... enjoy the feeling of superiority.

Time to re-evaluate our assumptions

First up for the day was the keynote speaker Jez Humble (TWITTER LINK), who spoke about a lot about his report "State of DevOps", in a talk called "What i learned from three years of Sciencing the crap out of Continuous Delivery". This was very interesting, and covered things that i had never thought about before, nor had even thought to question.

One thing that really stuck in my mind was the concept that if you "Move fast, you have to sacrifice quality". Yet all of the data he found was that actually you can have your cake and eat it too! You can move quickly, and you can actually be safer and more stable doing so. Something a lot of people don't believe, and i feel that people won't believe this unless they actually do it. It's something that sounds counter intuitive, but actually makes a whole lot of sense.

There was a few other items in his speech that were rather impressive. He found that employees who did non repetitive tasks required a workplace which allowed them to be free to make decisions and be creative, as well as the ability to make mistakes. Which, when you link in with the moving fast and the stability, all sort of fits together in a weird sort of way. If people are free to make mistakes, they can fix them. Being in a culture where you can mess up leads to better solutions, and hopefully prevents the mistake from ever happening. "Blameless Autopsies" is a term that was used a lot. Damn tech loves their funky terms.

Buy-in from Developers

One interesting point that kept coming up over and over again was the concept of "Developer Buy in". Simply put, developers need to be won over too. This may sound rather obvious, and it kinda is, but it's amazing the number of people and the number of teams who don't believe in the stories, concepts, and overall numbers behind how CD can improve things. Even Legacy systems can be updated / changed to work with CD, even without changing too much of the underlying business code. This was rather enlightening, and should excite a lot more people than It currently does.

Either way, i think there's a lot of work that needs to happen to sell CD to developers, and try and get them on board. The business unit won't push us, but other developers certainly can!

Metrics Matter

"Measure Everything, Not Just Production" was a talk from Steve Elliot, which was pretty much exactly what it said on the tin; MEASURE ALL THE THINGS!

With the likes of Logstash and ELK stacks becoming more prevalent in the overall development community, developers are getting more of an insight into their code and systems in real time. This is awesome, and it's something a lack of has hurt me in the past with personal projects. But not just live, but every system! Having the data to query later on once you have an idea is better than having an idea, and having no data to call on.

The flip side of the coin is that this data has to go somewhere, and sometimes too much "information" can hurt the overall visibility of your application. I guess theres a happy middle ground; having data AND using it is always better than not.

Testing

One talk that really got me thinking was Sally Goble's talk called "So What Do You Do When You Don't Do Testing?!". Short answer; move fast. Real fast.

Sally Goble works for the Guardian, and spoke about how that they basically gave up on Functional testing, as it resulted in no gain, and a whole lot of effort. Instead, if there's an issue with the site, they go ahead and fix it. Jobs done. Instead, they spend time finding ways to find bugs, and generally make "quality" better. This isn't to say they don't do unit tests. Basically, a term she coined was "Light touch testing". Know when to test, and when not to.

However, she did concede this method doesn't work for critical systems. Airplanes can't just update instantly, nor could a pace maker. You want to make sure that shits tested! So, it's knowing when to do what.

Communication is King

Communication matters a lot. Different parts of the business needs to open up to other parts, and in some cases the business needs a shuffle to ensure this can happen. The key communication needs to happen between your developers and your operations team. Without these two working together, you can't hope to succeed in CD. YOU NEED THESE TWO COGS WORKING TOGETHER! Ops needs to know what developers need achieve the business outcome. Developers need to know how to write code that will actually work on servers, and not BREAK EVERYTHING!

A rather obvious point i guess, but it's amazing that in 2016, with a plethora of ways to converse, we still SUCK at actually discussing things and sharing.

This wasn't from any particular talk, but instead a recurring theme from across the conference.

Time

I'm quite an impatient person. I don't like waiting around. But to move to CD does take time. You have to get the buy in, not just from the business by from Developers too. Ops and Developers need to change how they work and communicate, not just what code runs and where. Legacy code needs to be modified to fit this new concept. All of this takes time, money, and effort. I guess i was already aware of this, but it's eye opening seeing everyone going through the same motions, and knowing it's purely not just one place. It's part of doing CD.

Wrap up

Anyway, those are the main points i learned from PIPELINE CONF.There was so much to take in, that I'm sure I've missed a quite a lot. So much was shared, buy some really talented individuals. Highly recommend going to anyway, and i hope i get the chance to go next year! Thanks again to everyone who hosted and put PIPELINE CONF together. You people rock!! Also, here are some cool links:
Continuous delivery
State Of Devops report 2014

Using Twitter as a Microservice Orchestrator

So, that's a pretty weird title. How does someone use a Social Network to organise their Microservices?

To explain, let's jump back a few days.

I was having a chat to friend at work who was telling me about his project where he's set up a camera attached to his Raspberry Pi that watches his room while he's not there. However, he's not hooked it up so that that it turns off when he's home. The idea is to turn off the camera when his phone connected to the Wi-Fi - a pretty good indication of if you're home or not. Cool idea.

So, this got me thinking about how to turn this scenario into microservices. Well, the obvious breakdown would be to have a Network Poller Service that checks the network every so often for an IP address. It could then send a request to the Camera service to turn it off. Jobs done.

I was bored, and decided to see if I could create a similar network poller that scans my network at home, and then sends me a Direct Message on twitter every time someone connects or disconnects (you can check out the source code here).

However, this got me thinking; how do I alert twitter when the polling is finished? Do I just send the direct message right away? Or can I extract out this message process? And for that matter, can I make this process event driven?

So the idea of my architecture would look like this: Network Poller
(send Post request stating finished with data) -> Twitter Service
-> Camera Service (if I had one).

That makes sense. But wait, now our Network poller needs to know about both the Twitter Service, and the Camera Service. What happens if I add in another service?

This is where the event driven stuff happens; I need both the Twitter Service and the Camera Service to be listening to the Network Poller to make this work. So I actually need a 4th service hidden in here: Network Poller
-> Organiser
       -> Twitter Service
        -> Camera Service

Ok, so this new Organiser service should handle the change request the Network Poller sends, as well as then passing this information to the Twitter Service and Camera Service (who can then request the Network Poller Service if necessary for more info).

Now, I've got to build me an Organiser. Wouldn't it be great if a service already had streams, with a well-known API, and loads of different languages already supported? We do; Twitter.

In this process, we can actually use a bespoke Twitter Feed as the message stream for all of the services.

For example:

Network Poller posts a tweet onto the feed which is "#NP #updated" The Twitter Service is watching this, and when it gets this message it will ask the Network Poller for updated information, which it can then Direct Message me stating someone has either joined or left the network.

The Camera Service can ignore this, as it doesn't care about this tweet. But it could care about

"#NP #updated #master"

Which could mean that I'm home, and would then turn off the camera. Using certain hash tags or a certain string, you could send commands to different services on your network to do things.

My current thoughts for this format (I've yet to implement this in practise) are as such:

#ServiceIdentifier #verb #focus

Where "focus" is the focus of the message.

This would allow for multiple services to be looking out for a particular Tweets, without different services needing to know other services exist; they only care about the Twitter feed.

Draw backs

There are some pretty obvious draw backs to this idea.

To start with, you're relying on Twitter, a 3rd party company which you don't have control. Also, you're limited to 140 characters per message. Each message can't get too complex. If your apps are particularly chatty, you've got to watch out for usage limit too. Don't want to spam Twitters servers!

So, what do you think? Could it work? If you've got any thoughts or comments, go ahead and tweet me @Hazz223

The World Wide Web is basically the Royal Mail

Yep, you read that right! Arguably the greatest creation since fried chicken electricity pizza is basically the same as a couple of middle aged postmen. And no, the heat hasn't been getting to me.

Ok, hear me out here. By the end of this article you'll understand my crazy logic and maybe have a new found love of the British postal service*.

Let's get a look of the lay of the land

First up, I need to explain what the World Wide Web is. Annoyingly, people these days have decided that the term "Internet" and the "World Wide Web" is actually interchangeable. This is a load of balls. They're not the same, and in fact the World Wide Web is actually a subset. At this point, I'm pretty sure we in the tech business are just inventing names just to confuse the average Joe.

“So, how do they differ? How is the World Wide Web different? And why the shit do I care?"

Well, you care because you're on this site, and you want to have a rough idea on how the item you carry around in your pocket shares those irritating food based Instagram photos.

It's metaphor time (Not to be confused with morphing time)

How does one going about explaining the difference between the World Wide Web (fuck it, it's WWW from now on folks) and the Internet?

The WWW is basically a collection of computers, which when asked for information, they send back files, or more commonly known as Webpages. The internet is the underlying architecture on how it achieves this.

Annnnnnnnddddd that didn't help at all did it? Ok, it's far easier if you think of it like this; You write a letter to your other half asking for a nude photo that photo you look really good in as well as some other information. You post the letter. The post service delivers it to your other half. They send back information and photo. The post service once again takes it from their house to yours. And boom, you now have the information and photo you asked for!

That, my friend, is basically how the WWW works! Woo!!

Mail logo!

It's a little more complicated of course. Computers suck at understand us humans. So we have to be very explicit with what we ask for, and it has to be consistent. What we need (and of course use), is a protocol! A defined list of what we can and can't send, as well as defined format.

And here we have it: the Hyper Text Transfer Protocol (HTTP). This baby is how servers and your web browser understand each other. You'll probably have seen this in front of websites you visit.

So, how does this all link together with postage, the internet, and the WWW?

Putting it all together

Ok, so you sending consistent message to your friend, and with them returning in the same way each time, but with different content, is effectively how HTTP works. How does that link to postage?

When you post the message, it's picked up by a postman and is carried off to different centres and organised and all sorts. It's then taken to your friend's house where they read it and send back their response. This is very similar to how HTTP traffic is sent over the internet. You “address" it by using a URL (www.harrywinser.com), and away it goes! Across the internet. Through different servers. And eventually to its destination.

The internet is like the road under the postmen's feet. Without it, they'd fall into the pits of oblivion (or something). So, HTTP is the post, and the ground is the internet. HTTP sits on top of the internet. And in fact uses another protocol called TCP/IP. But I ain't getting into that now.

And actually, the internet isn't even what I defined above. It's not just the TCP/IP protocol. It's loads more. It's the concept of loads of machines connected together, all being able to communicate. Its giant weave of wires and technology, that even transcends this planet (see: satellites)! It's amazing that 100 years ago, it would take weeks to get a letter to the other side of the planet. Now it takes but a few seconds to share something.

Just think about that when you're looking at cat photos.

Oh, and apparently the internet weighs roughly the amount of a soft boiled egg. Yeah, someone worked it out. Yay.

*All lies. This article is all about the internet. Small side note, I actually when looking for a "postal appreciation day" and found NOTHING to do with England. Seriously guys, do we really hate our post that much?

An idea…

The purpose of this post is to share an idea that I've been bouncing around for the last few days. So far I've shared it with friends, and got a wide range of feedback. This has ranged from "Sounds like a great idea" to "Actually have you thought about this" to "Please sir, take your seat. I'm a bus driver; not your friend". So, I thought I should go ahead and share it on here, and see if I could gather some feedback.

The Idea

"What is this idea then?"
Simply put; it's a Dating app. Yep, another one. They seem to be a dime a dozen really. But what I'm trying to build is something a little different; you can't view a person's photos.

Ok, that's not strictly true. To view the other person's photos requires "work". Effectively, I want to get users learning about each other, before seeing each other. Here's how I plan to do it:

  • Person A searches (or is presented with), and finds Person B.
  • Person A has 3 questions given to them, such as "Do you like spicy food?" or "What's your favourite genre of film?" and so on. These questions have been decided by Person B
  • Person A gets all 3 correct
  • Persons B's photos have now been unlocked, with more profile information.

That's the kinda rough working. Of course, when Person A finds Person B they are given B's age, sex and location. They are also given a small Bio. Of course, this isn't the only way for A and B to get to know each other. Another way could be that A sends B a message, and once they've conversed a number of times, both A and B get the option to share profile : images and all.

Elephant in the room

Ok, so that's the basic premise. There is one obvious issue here; physical attraction matters quite a lot.
But after trying Tinder, I just felt sad that we've effectively cut down our interactions based solely on appearance. This is to combat this. This is the opposite. This is about a connection based on personality, before physical attraction.

So, feedback time

Please, I would like as much feedback as possible. If people think its rubbish; fine. If they would like to see a prototype; that's great. If you'd like to invest large sums of cash; let's talk. Basically; I want to know what you all think!!

I've shared this with a friend, and they thought it might work better as a Friend Finding app over a dating one; I'm now looking into modifying the premise. I just want as much feedback as possible before I start building!

Please tweet me, Email me, Facebook me, and/or Carrier Pigeon me your ideas. I want to see where this idea goes, and I really need your help to make it happen.

So, mull it over, and tell me what you think.

What is Code?

In many social situations, such as parties or at the doctors, it's customary for people to ask one other what they do for a living. When people ask me, I give the standard answer of "I'm a developer". This usually then leads to four different conversations;

  1. "Oh that's neat… Excuse me, I've left a turkey in the oven."
  2. "Oh cool! Hey, I've got [insert item of tech here]. What do you think? What should I buy next?"
  3. "Oh cool! Is that like the Binary thing? Like that film? The Matrix was it?"
  4. Mythical: "Oh yeah me too!"

The first instance I can't do much about. Some people are dicks. Sorry for breaking that to you.

The second point can range from being flattering, to downright annoying. I've been to parties (shut up, it happens) with people who ask constant questions and opinions on a laptop et al. Quite frankly if you're just going to be surfing the internet and receiving emails, then get what's cheapest… Or get that Mac you've just been on about for 20 minutes. Whatever. Case closed.

The fourth point does occur, and when it does it's a nice surprise, until we both mutually realise we're both as socially awkward as each other (Joking… That's only happened, like, 3 times).

The third point is the reason for this blog post. I'm going to attempt to explain what I believe code is, and what it means to me, without actually getting into writing code.

Class 101

Let's start with exactly what code is. Below is an example of a "Hello Word" in a language called Java*. This, usually, is what every programmer writes as their first test of a language. It's just standard, and there's probably a long history about it or something.

Hello world java

Anyway, first thing you'll notice is that IT'S IN ENGLISH! Not binary. Most programmers know binary, it's like an unwritten law. I've never needed while actually programming however.

Programming is effectively using a standard set of commands, and rearranging them and linking them together to form an overall output, such as a website or phone app. Programming languages are exactly what they are on the tin (???); they are languages. They have their own style, syntax, and concepts, just like English or Elvish. But, for the most part, they all achieve the same thing - communication!

You can write "Hello World" in almost any language, but how you do it can be entirely different. Take a look at this PHP:

Hello World php

Same result, but different syntax/style! Just like in traditional written languages. English for hello is...er... "Hello", while Spanish is "Hola" and Hawaiian is "Aloha". Same goal, different words.

It's a little like plumbing

So we now know computer code is a form of language, that's for the most part written in English, and different programming languages are formed of different syntax and styles but ultimately can achieve the same result.

Ok, onto the structure of code. Think of code as the plumbing in your house; there are different components working together to get water to and from your house. There are many different parts, and they are all connected together. Some parts are used when you turn the upstairs tap on, while others are used while showering, and so on. Some of these parts are specialised, other parts are duplicates such as bends etc.

It's all about knowing which part to use when and where. This is the same with programming. When do I use this bit, to achieve this output? And what can I do with this output, to continue to get something else that's more useful?

So how do different languages fall into this? Imaging different pipe material. They all achieve the same goal - moving one liquid to another spot - but are made out of different materials.

Actually, it's a bit like a waltz

Still with me? Hope so!

Here's where things get really complicated. Many people believe (and I am one of these people), that there is such a thing as elegant programming. This is different to efficiency. Efficient programming is akin to using the least number of pipes in your house that traverse the shortest distance. While good in theory, this leads your house to having ugly pipes exposed and nowhere to put your My Little Pony collection.

Elegant programming however is the idea that what you've made is easy to understand, obvious, clever, and easy to expand and maintain. It's a bit strange when you say obvious and clever in the same statement, but it's a thing. Promise.

This is where the concept of music comes in. Take a look at Twinkle Twinkle Little Star. It doesn't consist of many notes, and if you play it on a recorder, it sounds pretty basic and simple. But give the same tune to a master on, say a guitar, they could turn the very simple tune into a song that could easily be used to serenade couples down a boulevard. The concept of something being clever, yet obvious!

Music is the closest I can come to explaining exactly what code is. Music can be written down, and shared, but each musician can put their own twists on it. They can also change the instruments. Add more in. Take some out. It's a mixture of maths and soul. Logic and heart. And that's exactly what programming can be. You can build a huge system that all works together to produce a product (an orchestra playing a symphony), or a tiny program that does one thing but well (guitar busker).

This concept is something that I love about my work. It's something that gets me programming during my weekends, and gets me pumped when I have a new idea. This concept of building something that produces a useful product, but is elegant and beautiful in its own right. Even if no one but me sees it.

Anyway, I hope you enjoyed my introduction into what programming is. That's all for this week folks!

*Related to that constant pop up on your computer asking to update. The other one to Adobe Flash.