Tag Archives: Open Source

Creating RR Records in Route53 with Ansible

So if you are like me and using Ansible to deploy all of your infrastructure, you probably run into more than your fair share of issues. Often Ansible modules take a list or a string, but output fancy structured data. One particular case of this is if you are using the route53 module and creating RR records in DNS. Here is what the route53 module accepts as a value:

- value
        The new value when creating a DNS record.  Multiple comma-
        spaced values are allowed.  When deleting a record all values
        for the record must be specified or Route53 will not delete
        it. [Default: None]

So here you can see that it takes a string (the source code indicates it also accepts lists) listing the ip addresses to point to. If you are deploying in a cloud however you probably need to pull the IP addresses from the inventory, which is stored as complex structured data. Extracting a list of IP addresses out of this is an extremely difficult task. To do this we will need to make heavy use of the features Ansible provides us through its Jinja2 variable expressions.

The first thing that we need is a list of all the variables for the machines in a particular group. Ansible doesn’t provide this, but it does provide two similar variables. First, it provides a variable names ‘group’ which gives the names of all the members of a particular group. Secondly it provides the ‘hostvars’ variable which gives you all of the variables in a dictionary with keys as the hostname and the variables as the values. As there is no builtin that does what we want we are going to need a custom filter plugin.

To install this plugin place it in a directory then add that directory to the end of your filter_plugins configuration option. For example if you placed the file in the plugins/filter directory your filter_plugins line might look like this:

filter_plugins     = /usr/share/ansible_plugins/filter_plugins:plugins/filter

This plugin will take a dictionary and a list, lookup all the keys within the list and return their values in a new list in the same order as the keys were in. So we can use this to get all the variables for the machines we are interested in. For example to get the variables for the hosts in the webservers group we can now do this:

{{ hostvars|fetchlistfromdict(groups.webservers) }}

Now that we have the right variables we simply need to filter it down to just the attribute we need. Jinja2 provides the map filter for this purpose. Map takes a list and can either pick out a particular attribute, or run another filter on all the items in the list. So to get the IP addresses we would do this:

{{ hostvars|fetchlistfromdict(groups.webservers)|map(attribute='ansible_default_ipv4.address') }}

If you try to print this out in a debug line you might notice that you don’t get a list in response and Ansible instead gives you a string with the following:

 <generator object do_map at 0x2d7f640>

To work around this issue you need to explicitly convert it to a list. This is fairly simple with the list filter. Adding that onto this example looks like this:

{{ hostvars|fetchlistfromdict(groups.webservers)|map(attribute='ansible_default_ipv4.address')|list }}

You now have a list of all the IP addresses ready to pass into a template, a role, or in this case into the route53 module. Finally we put this all together to create that round robin record we wanted at the start.

route53:
  command: create
  zone: example.com
  record: rr.example.com
  type: A
  value: "{{ hostvars|fetchlistfromdict(groups.webservers)|map(attribute='ansible_default_ipv4.address')|list }}"

Resources

Ansible Talk @ Infra Coders

Here are the notes that I used in my talk at Infrastructure Coders. Each section was also put on the screen as a ‘slide’. The configuration that I used in the demo is available at GitHub. A full video of the meetup is available on the Infrastructure Coders Youtube Channel, my talk starts at 25:05.

Ansible
--------

0. There is nothing in the hat
  - Start a RHEL install
    - Cmd line: console=ttyS0 ks=http://admin01/ns3.cfg
    - If you want to follow the demos grab the ansible config from my github
    - You will need to substitute hostnames in the ansible hosts file
    - You should copy the firewall config from ns2 (remove port 647 if paranoid)

1. The problem
  - Ansible is the combination of several functions
  - There was a plan to build config management on func
  - However func is a pain to setup
  - Puppet and Chef have a steep learning curve
  - Ansible was also built to simplfy complexrelease procedures
  - You need to know ruby to extend Puppet/Chef

2. Ansible
  - Designed so you can figure out how to use it in 15 minutes
  - Designed to be easy to setup
  - Doesn't require much to be installed on the managed host
  - Designed to do config management/deployment/ad hoc
  - Other people do security better, just use SSH
  - You can extend ansible in any language that can output JSON

3. Simple Ansible Demo
  - Ansible hosts file
  - Ansible can be run directly on the command line
    - Run cat /etc/redhat-release
    - Get info using the setup module
  - It can prompt for auth, or use key based auth
    - On the new machine show it prompting
    - Run the rhelsetup script on the new machine
    - Install vim-enhanced

4. Playbooks
  - This is the method of scripting Ansible
  - Done in YAML
  - Executed in order *gee thanks puppet*
  - Designed to be as easy as possible

5. Example playbook
  - Playbook for the name servers
    - https://github.com/smarthall/tildaslash.com/blob/master/playbooks/zones.play
    - Can have multiple plays in a book
    - Can serialise if you dont want all to be down at once
  - Template config for the name servers
    - https://github.com/smarthall/tildaslash.com/blob/master/playbooks/zones/named.conf.j2
  - Firewall install script
    - https://github.com/smarthall/tildaslash.com/blob/master/playbooks/firewall.play

6. My thoughts
  - Config management has been around a while, its going from art to science
  - Ansible covers more ground than puppet and chef do
  - Ansible doesn't compromise on simplicity to do that
  - I don't have to focus on the nodes, I can focus on services
  - There is something missing
    - Disk config is done in kickstarts
    - Network config can't be done by Ansible
    - Need to find a way to cover both with one

The playlist of all the videos is available at Youtube.

mod_pagespeed is not (always) the answer

What is mod_pagespeed

Google recently released a chunk of code in the form of an Apache module. The idea is that you install it in your Apache server, it sits in between your application and the web browser and modifies the served requests to make the page load faster.
It does this by using combinations of filters, some are well known best practices, others are newer ideas. For example on filter simply minifies your JavaScript while another embeds small images in a page using data-uris. The changes these filters make range from low risk, to high risk. It should be noted that not all the filters will improve the page time some even making pages slower in some cases.

So what’s the issue?

The issue here really isn’t mod_pagespeed, but it’s the way people are viewing it. In my job as a Web Performance Engineer I have had several people recently say to me “let’s put mod_pagespeed on our web server to make it faster”. This is a break from normal attitudes, if someone were to to say “we should put our images into data-uris” then people would question the speed benefit, or the extra load on the server. For some reason when Google implement a page speed module people just assume that it will make their page faster, and that it will work in their environment. The truth is that Google really have no idea what the module will do to your page.

The second issue is that all these tweaks can usually be better implemented at the application level. If you minimize all your JavaScript as part of your build process then the web server will not have to do it for you. The same applies to data-uris. If they are simply part of the page then the browser doesn’t need to read in the extra image, uuencode it, then compress it. All that is quite a lot of work, which only really needs to be done once.

So what should I use mod_pagespeed for then?

You don’t always have access to the application code. If you are using third party software then before mod_pagespeed you may have had no control over the minification of CSS. This is where the module really shines. It gives you a layer between the application code and the web browser where you can apply all sorts of performance tuning.

The other advantage I can see is for looking for the best tunings to apply to your application quickly. You can setup mod_pagespeed and and run experimental tests with the filters on of and with a control to quickly figure out what rules you should apply in your application.

Google G1: Six Months On

So six months ago I bought my Google G1, my first impressions were excited and extremely positive. Has this phone stood the test of time though?

Physically

The phone is still in good physical condition, which is more than I could have said about my old XDA Atom Flame after six months. There are a few scratches on the screen, but I bought a screen protector for it so I can simply peel them off. Surprisingly the various crevices on the phone have avoided build ups of dust which commonly plagues my phones. The battery is beginning to fade, and can only last me around 12 hours with my ordinary usage (which is probably considered heavy usage). This makes weekends away from home interesting as I have to avoid using my phone to stretch the battery over 24 hours.

When I first got the phone I expected that the keyboard keys would fade, or that the keyboard snap mechanism would somehow break. I was wrong, the keys are still as visible as when I first got it, and the snap mechanism still works perfectly.

The OS

In the time I’ve had this phone Android has gone from 1.1 to 2.0. Sadly there haven’t been any official new releases of the phone software. There have however been releases of the well known mod for this phone called ‘CyanogenMod’. Currently CyanogenMod is at Android version 1.5 with parts of 2.0 ported across.

Since the first week I had the phone I’ve been using CyanogenMod and have seen the improvements in it take it from strength to strength. Originally it looks almost the exact same as the original OS but now it includes several features that I could not live without. My favorites would be:

  • Tethering to my Linux PC
  • OpenVPN settings
  • 360 degree rotation
  • Improved contacts screen with direct call links
  • Voice Search

The Applications

Like any mobile OS the best part is the applications. This is where an OS either make it or breaks it. While Google have been constantly improving the Android platform old apps have remained around and stayed compatible with the phone. Google has also held two developer competitions during the time I’ve had the phone which has brought loads of new apps and innovation. So as each application is its own entity I’m going to review my favorites separately.

Google Maps

When I got the phone Google Maps was simply a map, with limited search capability and able to give directions. Since then however Google have added Street View, Navigation (US Only sadly), Buzz and much better searching. For something I used once a month I now use it almost daily.

ConnectBot

One of the reasons I went for a phone with a hardware keyboard was to make SSHing into my Linux machines easier. ConnectBot handles this perfectly. I cannot stress enough how useful this application is. Recently it has been improved to include support for SSH agents too which improved things even further.

My Tracks

As someone who enjoys hiking and walking having a GPS logger can be extremely useful. My Tracks basically turns your Android phone into a GPS logger and displays the data for you on a map. It also allows you to export the logs in popular formats or simply upload them to My Maps on Google. It can also graph your elevation, speed and display interesting statistics.

Conclusion

All up I still enjoy this phone, and still use it daily. I am looking at moving to either an N900 or the Google Nexus One next. I haven’t moved because the N900 has been having trouble with the USB connectors breaking off, and the Nexus One is too expensive to import into Australia. I doubt I’ll be moving to another phone any time soon and this phone doesn’t look like it will give out any time in the near future.

Random Thought: What is the cell phone market going to look like five years from now? And where the hell is my wristwatch phone?