Tag Archives: Programming

Creating RR Records in Route53 with Ansible

So if you are like me and using Ansible to deploy all of your infrastructure, you probably run into more than your fair share of issues. Often Ansible modules take a list or a string, but output fancy structured data. One particular case of this is if you are using the route53 module and creating RR records in DNS. Here is what the route53 module accepts as a value:

- value
        The new value when creating a DNS record.  Multiple comma-
        spaced values are allowed.  When deleting a record all values
        for the record must be specified or Route53 will not delete
        it. [Default: None]

So here you can see that it takes a string (the source code indicates it also accepts lists) listing the ip addresses to point to. If you are deploying in a cloud however you probably need to pull the IP addresses from the inventory, which is stored as complex structured data. Extracting a list of IP addresses out of this is an extremely difficult task. To do this we will need to make heavy use of the features Ansible provides us through its Jinja2 variable expressions.

The first thing that we need is a list of all the variables for the machines in a particular group. Ansible doesn’t provide this, but it does provide two similar variables. First, it provides a variable names ‘group’ which gives the names of all the members of a particular group. Secondly it provides the ‘hostvars’ variable which gives you all of the variables in a dictionary with keys as the hostname and the variables as the values. As there is no builtin that does what we want we are going to need a custom filter plugin.

To install this plugin place it in a directory then add that directory to the end of your filter_plugins configuration option. For example if you placed the file in the plugins/filter directory your filter_plugins line might look like this:

filter_plugins     = /usr/share/ansible_plugins/filter_plugins:plugins/filter

This plugin will take a dictionary and a list, lookup all the keys within the list and return their values in a new list in the same order as the keys were in. So we can use this to get all the variables for the machines we are interested in. For example to get the variables for the hosts in the webservers group we can now do this:

{{ hostvars|fetchlistfromdict(groups.webservers) }}

Now that we have the right variables we simply need to filter it down to just the attribute we need. Jinja2 provides the map filter for this purpose. Map takes a list and can either pick out a particular attribute, or run another filter on all the items in the list. So to get the IP addresses we would do this:

{{ hostvars|fetchlistfromdict(groups.webservers)|map(attribute='ansible_default_ipv4.address') }}

If you try to print this out in a debug line you might notice that you don’t get a list in response and Ansible instead gives you a string with the following:

 <generator object do_map at 0x2d7f640>

To work around this issue you need to explicitly convert it to a list. This is fairly simple with the list filter. Adding that onto this example looks like this:

{{ hostvars|fetchlistfromdict(groups.webservers)|map(attribute='ansible_default_ipv4.address')|list }}

You now have a list of all the IP addresses ready to pass into a template, a role, or in this case into the route53 module. Finally we put this all together to create that round robin record we wanted at the start.

route53:
  command: create
  zone: example.com
  record: rr.example.com
  type: A
  value: "{{ hostvars|fetchlistfromdict(groups.webservers)|map(attribute='ansible_default_ipv4.address')|list }}"

Resources

Simple Configuration Service

The Basics

So I’ve seen a few implementations of ‘config services’ being built in all sorts of {ruby,python,php,java} applications. It really gets on my nerves to see people implementing things that could be much simpler. As Engineers we should really strive to get the most simple product built in a flexible way. So here is my go at implementing a ‘config service’ (basically a key value store optimised for reads). This was originally tweeted in a set of four tweets (1, 2, 3, 4).

Tweet 1

A 'Config Service' that fits in a tweet:
<Location /config>
  DAV svn
  SVNPath /var/svn/config
  SVNAutoversioning on
</Location>

Simply place this in your apache server configuration. You’ll need to create the SVN repo, which can be done with ‘svnadmin create /var/svn/config’ then chmod it to be writiable and readable by apache. What we are doing here is turning on WebDAV (which lets us upload and make directories on the apache server) and turning on SVN backed autoversioning which means that every change we make will be recorded in the SVN repo.

Tweet 2

To get a variable:
curl -s http://config/config/namespace/key

This simply performs a GET request for a key. You can’t do this until you add some keys to the server.

Tweet 3

To change a variable:
echo "value" | curl -T- http://config/config/namespace/key

This creates/modifies a key. You will need to have the namespace created first, which is in the next tweet.

Tweet 4

Make a new namespace:
curl -XMKCOL http://config/config/namespace

This creates a namespace to put keys in.

Extensions

Users

The service above doesn’t support users, so anyone can modify the keys, and the svn logs look like this:

------------------------------------------------------------------------
r5 | (no author) | 2012-10-15 12:36:33 +1100 (Mon, 15 Oct 2012) | 2 lines

Autoversioning commit:  a non-deltaV client made a change to
/namespace/key
------------------------------------------------------------------------

All we need to do is require the user to authenticate when doing PUT requests to the server. You can use this method to restrict certain namespaces too. Generate a htpasswd file for the users and place it in /var/svn/users and change the code to read as follows.

<Location /config>
  DAV svn
  SVNPath /var/svn/config
  SVNAutoversioning on
  AuthName "Config Service"
  AuthType Basic
  AuthUserFile /var/svn/users
  <LimitExcept GET>
    require valid-user
  </Limit>
</Location>

Viewing the logs

Simply do this:

svn checkout http://config/config
cd config
svn log

You can also use this to make bulk/atomic changes in the same way you would make changes to a subversion repository.

Programming In Javascript

Javascript is an interesting language. Its partly a functional programming language and part object oriented. It uses a C style syntax but borrows its naming conventions from Java (mostly). Personally I find Javascript language to be one of the most interesting languages that I have played with. The complaints I hear most often regarding Javascript are that it is very hard to learn and that there are many subtle differences between the interpreters.

Difficult to learn

This used to be mostly true. Javascript was a poorly documented language, often only documented in tutorial form by w3schools, or technically documented as ECMAScript. The absolute wealth of tutorials and blog posts made the good information few and far between. Largely when looking for information on how to perform a particular function you had to download some sample code and figure out how it was done based on that.

More recently though Javascript has caught the wave that is trying to standardize the web and this has somewhat improved the situation. Browser manufacturers are documenting their Javascript implementations and largely converging on a common standard. Additionally many helper libraries have been introduced to make the task of working on Javascript even easier. Once Javascript may have been difficult to learn, but as of late this is no longer true.

Javascript Documentation:

Subtle differences in interpretation

This is is one of the biggest problems you still see in Javascript today. You will often find developers writing functions to simply deal with the differences between browsers, there are even entire libraries dedicate to to abstracting away the differences. If I had a dollar for every implementation of a function to get a XMLHttpRequest object across browsers, I wouldn’t need my job.

Unfortunately it is still however very important to know the differences between implementations of Javascript if you plan on writing anything that will run on more than one browser. These difference may be in the features available in the language, in the Document Object Model or in the way the browser handles CSS. Thankfully many people work on documenting the difference and abstracting around them in libraries.

Javascript Implementations Differences:

Javascript Libraries:

Random Thought: If only Facebook didn’t get in the way of Javascript all the time…

Writing a Daemon in C

What is a Daemon?

A daemon is a program that runs in the background. A daemon will usually be started at system startup and end at system shutdown. The exceptions to this rule are programs like the Bluetooth SDP daemon, which is activated when a new Bluetooth HCI is found,, and ends when it is removed. Daemons run transparently and do not normally interact with the user directly.

Daemons start as ordinary processes but they eventually ‘fork and die’ to start running in the background. Some daemons do only the ‘fork and die’ step but ignore other important steps. Here is a list of what a daemon should do:

  1. Fork to create a child, and exit the parent process.
  2. Change the umask so that we aren’t relying on the one set in the parent.
  3. Open logs to write to in the case of an error.
  4. Create a new session id and detach from the current session.
  5. Change the working directory to somewhere that won’t get unmounted.
  6. Close STDIN, STDOUT and STDERR.

These steps ensure that our association with the calling environment is destroyed and our daemon is now free to run as a completely separate process.

Lastly before writing the daemon you should make sure the code is written securely and in a way that fails gracefully. If your daemon crashes it will not be able to prompt the user about what action to take. The user may not even notice until it is too late.

Forking a child process

In Unix fork() is the only system call with two return values. When you call fork a child process is created which is a near copy of its parent (some things will be different in the child eg. process id). The fork command then returns a 0 in the child and the childs process id in the parent, on failure a -1 is sent to the parent. Generally a program will then check whether it is the child or parent by these return values (just like in movies when a cloned character will check to see if he has a belly button and hence is the original). Here is a snippet of code to do this:

pid_t pid;

/* Clone ourselves to make a child */   
pid = fork(); 

/* If the pid is less than zero,
   something went wrong when forking */
if (pid < 0) {
    exit(EXIT_FAILURE);
}

/* If the pid we got back was greater
   than zero, then the clone was
   successful and we are the parent. */
if (pid > 0) {
    exit(EXIT_SUCCESS);
}

/* If execution reaches this point we are the child */

Changing the umask

Because we are a clone of our parent we’ve inherited its umask. This means the child doesn’t know what permissions files will end up with when it tries to create them. We do this by simply calling umask like this:

/* Set the umask to zero */
umask(0);

Open logs to write to

This part can be done in several different ways. You could open text files, log to a database or use syslog. The method I’m going to demonstrate here is to log using syslog. Syslog sends your log messages to a system wide logger, where they can be configured to be written to a file, send to a network server or filtered away entirely.

/* Open a connection to the syslog server */
openlog(argv[0],LOG_NOWAIT|LOG_PID,LOG_USER); 

/* Sends a message to the syslog daemon */
syslog(LOG_NOTICE, "Successfully started daemon\n"); 

/* this is optional and only needs to be done when your daemon exits */
closelog();

Create a new session id

Each process on a Unix system is a member of a process group (or session). The id of each group is the process id of its owner. When we forked from our parent earlier we will have inherited its process group, and our process group leader will still be its parent process. We want to create our own process group and become our own process leader otherwise we will look like an orphan. We can do this easily as follows:

pid_t sid;

/* Try to create our own process group */
sid = setsid();
if (sid < 0) {
    syslog(LOG_ERR, "Could not create process group\n");
    exit(EXIT_FAILURE);
}

Changing the working directory

At the moment we have the working directory we inherited from our parent. This working directory could be a network mount, a removable drive or somewhere the administrator may want to unmount at some point. To unmount any of these the system will have to kill any processes still using them, which would be unfortunate for our daemon. For this reason we set our working directory to the root directory, which we are sure will always exist and can’t be unmounted.

/* Change the current working directory */
if ((chdir("/")) < 0) {
    syslog(LOG_ERR, "Could not change working directory to /\n");
    exit(EXIT_FAILURE);
}

Closing the standard file descriptors

A daemon doesn’t interact with the user directly it has no use for STDIN, STDOUT and STDERR and we really have no idea where these are connected or where anything we write to them will end up. As these file descriptors are not required and effectively useless we should close them to save some system resources and prevent any related security problems. We close these descriptors like this:

/* Close the standard file descriptors */
close(STDIN_FILENO);
close(STDOUT_FILENO);
close(STDERR_FILENO);

Writing the payload

Now you have a C program that is capable of becoming a daemon, but its a pretty useless daemon if it exits immediately. Payload code is really up to you to design. I’ll offer you a few tips on designing your payload.

  • Put your payload in a loop. Generally in a daemon you want to perform the same action over and over again until you’re killed. If you have to cleanup (such as closing syslog) when the daemon is about to be killed you should add an exit clause that will be activated by a SIGTERM signal handler.
  • Make your code as fast an efficient as possible. This is something you should do with any program, but with daemons it is important that you do not hamper the performance of the rest of the system. This is especially true if you’re going to be running this daemon on desktop systems.
  • Be aware that your code may be preempted very often. As your daemon is going to be running for the amount of time the system is up, it is likely that its execution will be preempted.
  • Be paranoid about security. Daemons are common attack vectors and can be used to gain privileged access to a system. You should consider dropping any privileges that you don’t require.

Conclusion

So if we take all the code I’ve mentioned in this post and put it all together you have a simple daemon. You can download the source from the link here: daemon.c.
If your daemon is only going to be run on Linux and not on a System V style system such as Solaris you can use the daemon function to do a lot of this work for you.

References

Linux Daemon Writing HOWTO in C
Linux Daemon writing in C++

Random Thought: It appears the devil uses a Unix based OS, probably OSX.

Paramaterized Java Classes

One of the biggest features of Java 1.5 was generics. In particular all the collection classes had been extended to use parametrized classes. Normally the collection classes accepted and returned Objects which is the class all other Java classes descend from. Unfortunately this meant that you had to cast everything you got back out of a collection to what you expected it to be. and until you did you would only be able to call methods that were provided by Object. You also had to be ready to catch an exception in case the class could not be cast because it was the wrong object.

Generics and parametrized classes allow Java programmers to place a type on a class and have that type inherited by its methods. For example you can now declare an ArrayList class with a type String. This alters the ArrayList class so that its add method now only accepts objects of type String, the get method now also returns objects of type String. This makes everything type safe which means you don’t have to cast anything and your code won’t compile if you try to put something in the ArrayList that doesn’t match its class.

Java uses parametrized classes to build its collections and you’ll want to use them too if you’re making your own collection class. For example if you were implementing a stack, a queue or a multi-priority FIFO queue are good cases for parametrized classes. Be careful though of the lure parametrized classes can have. They are not a replacement for polymorphism and shouldn’t be used when polymorphism would make more sense. For example if your multi-priority queue gets the priority out of the object itself then you’d need an interface that provides a method to get the priority. Then your class will only be able to accept items that implement that interface, which makes sense in this case as we need to priority to be able to store it.

A parametrized class is really simple to use. Here is an example implementation of a stack collection backed by an ArrayList:

import java.util.ArrayList;
import java.util.Collection;

/**
 * This class acts as a stack. Items can be 'pushed' which adds them to the top
 * of the stack. items can also be 'popped' which removes and returns the top
 * item on the stack and removes it. This means only the most recently added
 * item is available at the current time. To get to older items you need to
 * first remove the others.
 * 
 * Note: Java already has a stack object that should probably be used in
 *       preference to this one. This is only an example implementation.
 * 
 * @author Daniel Hall <daniel@danielhall.me>
 *
 * @param <T> The type of items that can be stored in the Stack.
 */
public class Stack<T> {
	/* Uses the same type as this class to store the items */
	private ArrayList<T> array = new ArrayList<T>();
	
	/**
	 * Creates a Stack containing items already in a collection. The collection
	 * must have the same parameterized type as this class to ensure that we get
	 * the right objects.
	 * @param c The Collection to initialize with
	 */
	public Stack(Collection<T> c) {
		array.addAll(c);
	}
	
	/**
	 * Creates an empty Stack object
	 */
	public Stack() {
		
	}
	
	/**
	 * Adds an item to the top of the stack.
	 * @param item The item which will be added to the top of the stack.
	 */
	public void push(T item) {
		array.add(item);
	}
	
	/**
	 * Removes the first item from the stack
	 * @return The item that was on the top of the stack.
	 */
	public T pop() {
		/* This gets the size so we don't have to do it twice. */
		int count = array.size();
		
		/* If the stack is empty return null, note that the Java implementation
		 * of stack throws an Exception instead.
		 */
		if (count == 0) {
			return null;
		}
		
		/* Remove the last added object (which will have index count - 1) */
		return array.remove(count - 1);
	}
}

Random thought: John Lions wrote a book about the Unix source code, in the seventies, which because it also included some code, was blocked from being published until 1996.