Links: 4-27-2015

  • Everything We Wish We’d Known About Building Data Products – First Round Review
    Quote: "Where to Start Building: A lot of people choose to start building by modeling the product in question. Some start with feature discovery or feature engineering. Others start with building the infrastructure to serve results at scale. But for Belkin, there’s only one right answer and starting point for a data product: Understanding how will you evaluate performance and building evaluation tools. “Every single company I’ve worked at and talked to has the same problem without a single exception so far — poor data quality, especially tracking data,” he says.“Either there’s incomplete data, missing tracking data, duplicative tracking data.” To solve this problem, you must invest a ton of time and energy monitoring data quality. You need to monitor and alert as carefully as you monitor site SLAs. You need to treat data quality bugs as more than a first priority. Don’t be afraid to fail a deploy if you detect data quality issues."
    (categories: data quality testing bigdata linkedin twitter information )

Links: 4-22-2015

  • Meet DJ Patil: Obama’s Big Data dude
    Quote: "In each new feature, Patil drove home the idea that the best sign of a good data product is no obvious evidence of the data itself. “The user doesn’t want to see raw data, they want the data in a usable form, and that usable form should help them do something more creative, be more efficient, give them superpowers,” he said. “Something you could never conceive of before.”" What a great story too.
    (categories: data visualization value problem-solving grit government )

  • A Conversation with Werner Vogels – ACM Queue
    On services before microservices were a thing. Also note that this was written in 2006(!) and that Jim Gray (since lost at sea) did the interview: "The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. This customer feedback loop is essential for improving the quality of the service."
    (categories: amazon soa microservices wernervogels )

Links: 4-14-2015

Links: 4-8-2015

Links: 4-7-2015

Links: 4-6-2015

  • 4 reasons why microservices resonate – O’Reilly Radar
    Quote: "One of the reasons we talk about “microservices” instead of just “services” is the unit of change. Traditional service-oriented architectures are typically designed to maximally leverage shared resources. The side effect of a shared resources design is a lot of physical coupling, where each service shares a monolithic database, an object-relational mapping layer, and lots of shared implementation elements. Architects are also drawn to single sources of truth to eliminate duplication, which encourages “smart” architectural elements like Enterprise Service Buses to handle chores like transformation and orchestration. If you look at SOA architectural goals, the desire is to encapsulate behavior and secondarily share resources. However, traditional SOA architectures make isolated change hard: when everything is highly coupled at the implementation level to share resources effectively, that coupling hinders isolated change. Microservices strive for integration coupling but no physical coupling. The goal is both encapsulated behavior and ease of operational change."
    (categories: microservices soa architecture devops )

  • Meat Production Wastes Natural Resources | Animals Used for Food | The Issues | PETA
    It takes more than 2,400 gallons of water to produce 1 pound of meat, while growing 1 pound of wheat only requires 25 gallons. You save more water by not eating a pound of meat than you do by not showering for six months!
    (categories: energy water food )

Docker, Minecraft and Bukkit…err CanaryMod

Last Friday was a “Spring Holiday” at Jive but all my little dudes were in school so I figured it’d be a great day to go into work and actually try to do something technical for a change of pace. My oldest has become completely absorbed in Minecraft and offered up that he’d really like to learn how to program mods a couple weeks ago so we bought the Kindle version of this book, which suggested the use of Bukkit. Simultaneously, I really wanted to get my hands a teensy bit dirty with Docker so I spent the time on Friday trying to get the two to place nicely together so that I could deploy a Docker image of Bukkit to my hosted server. Sidenote: I’m well aware of the fact that there are tons of Docker images already built for Minecraft, I personally learn faster and better by doing rather than reading.

My personal laptop is running OSX so the first order of business was getting Docker installed there. Docker doesn’t actually natively run on Macs so I installed boot2docker, which I think is a little bit Inception-ish (VM inside of a VM?) but seems to work just fine. After that I ran through this tutorial:

https://www.docker.com/tryit/

which was / is really well done. Not sure if there’s a tool that they used to build the interactive console but either way, great work on their part. I got my Docker.io account setup (https://hub.docker.com/u/ajohnson1200/) and then started down the path of trying to create a Docker image for Bukkit.

As I was investigating Bukkit as a development server I came across this presentation:

which forced me to call an audible at the line, didn’t look like Bukkit was going to be a valid platform to develop on. Turns out that the book I mentioned above has a second version, which instead of using Bukkit, uses CanaryMod, which does allow for custom Java plugins and doesn’t have licensing problems like Bukkit.

After going through the tutorial (which was really basic), my first thought / approach was to spin up a simple image so I started using this one:

https://registry.hub.docker.com/u/tianon/true/

and then using the console to install packages into that and to then save and publish it. But that was a rookie move, Docker has a much better way of creating images via Dockerfile, where you write a recipe for the image that you want to create and then let Docker build that from the recipe. So then I spent a bunch of time looking at some of the existing Minecraft (Bukkit or CanaryMod) Docker images that exist in the wild, examples:

https://github.com/mbrevoort/docker-minecraft-bukkit

https://github.com/tclavier/docker-scriptcraft

https://github.com/chrisabrams/docker-minecraft-with-bukkit

all of which were helpful. Became obvious that I’d use a Dockerfile and some kind of startup script and so after thinking about everything for a bit I ended up with a Dockerfile that looks like this:

from dockerfile/java:oracle-java8
env DEBIAN_FRONTEND noninteractive

add canarymod.jar /opt/minecraft/
add start /start
run chmod +x /start

volume ["/minecraft/"]
cmd /start

First line indicates that I want to base my image on the Java8 image (right?), second line puts the OS into noninteractive mode as documented here. Next I take the jar file that I downloaded from the CanaryMod site and add it to /opt/minecraft and alias my startup script to /start and give it the proper permissions. I create a mount point using the volume instruction which I’ll use later as the place that CanaryMod will start it’s data and then finally a cmd instruction to initialize the container, which leads to… a startup script that looks like this:
#!/bin/bash
set -x
set -e
cp /opt/minecraft/canarymod.jar /minecraft
cd /minecraft

/usr/bin/java -Xmx4096M -jar canarymod.jar
First two lines I copy / pasted from another startup script without knowing what they did but just read up on them via this nice blog post so basically we have some nice defaults for the startup script. Next, I copy the jar from the location that it exists in the image into the volume defined in the Dockerfile and then startup the app on the last line. When CanaryMod starts up, it’ll automatically create a bunch of directories and a couple of files:

canarymod.jar config db dbadapters eula.txt lang logs pluginlangs plugins usercache.json worlds

one of which you’ll need to change (eula.txt) before it’ll complete the startup process. I left that as a manual process that you’ll need to do if you use my Docker image (which you shouldn’t, mine is very basic). Once I had those written I built the image:
docker build -t mycanaryapp .
and then I decided that I didn’t like the name so I changed the name:
docker tag mycanaryapp:latest ajohnson1200/canarymod:latest
and then I pushed it to the Docker repo:
docker push ajohnson1200/canarymod
After that, I logged into my hosted server (which has been running for years on Rimuhosting) and attempted to install Docker. Docker wouldn’t work with the kernel I had installed so I had to update that and then got Docker installed in a couple minutes, pulled my image:
docker pull ajohnson1200/canarymod
and then started up the container:
docker run -v /usr/hosts/canarymod:/minecraft -p 25566:25566 --rm ajohnson1200/canarymod
binding the external port 25566 to the same internal port since I another Minecraft server already running on 25565 although now that I think about it, I think I could run CanaryMod inside the container on 25565 and just bind the external port to 25566 and not have to change the config that lives inside /minecraft. I’ll have to check that out.

Not sure if I took the right approach or not but the big thing that I didn’t spend time thinking about was storing data inside of the container instead of persisting the data on the volume outside of the container. I *think* that they way I’ve done it actually makes it possible to continually update the CanaryMod distribution while not touching the data that it stores, which is nice.

All in all, a fun exercise and I learned a bunch. If you’re looking for a CanaryMod image, don’t use mine. :)

Other links:

Now with 50% less caffeine!