How To Manage Your Servers With Rex - Best Practice

http://www.howtoforge.com/how-to-manage-your-servers-with-rex-best-practice


(R)?ex is a server orchestration and configuration management tool. With (R)?ex you can manage all your boxes from a central point through the complete process of configuration management and software deployment.
In short words, Rex is like Make. There is a central Rexfile where you can define tasks. These tasks gets executed on remote machines over ssh. The tasks are written in plain perl.
You can get (R)?ex from its website http://rexify.org/.

Preface

In this guide I will use Subversion to manage all the Tasks. You can use any other SCM system as well, as long it supports something similar to Subversion's external directive. I'm using Ubuntu 12.04 but you can use other distributions as well.
I won't append "sudo" on every command, please use "sudo" where it is applicable.
In this guide I will create two example projects. One project is named 'website' and an other called 'database' because in larger companies there is often a split between system administrators and database administrators. Both projects will use the "common tasks" that can be managed for example from a central operations team.
I will use multiple servers:

• Subversion server, svn01
• Database Server, db01
• Web Server, web01
• Workstation, wks01

Set Up Code Repositories

First your have to install all the needed packages. Execute this command on the Subversion server.

svn01# apt-get install libapache2-svn subversion apache2-mpm-prefork

Now edit the file /etc/apache2/mods-enabled/dav_svn.conf and paste the following code into it (replacing existing content).

  DAV svn
  SVNParentPath /var/lib/svn
  AuthType Basic
  AuthName "Subversion Repository"
  AuthUserFile /etc/apache2/dav_svn.passwd
  
    Require valid-user
  

Now create the directory /var/lib/svn and all the needed repositories. I will describe the contents of the repositories later.

svn01# mkdir /var/lib/svn
svn01# cd /var/lib/svn
svn01 /var/lib/svn# svnadmin create common
svn01 /var/lib/svn# svnadmin create service
svn01 /var/lib/svn# svnadmin create database
svn01 /var/lib/svn# svnadmin create website
svn01 /var/lib/svn# chown -R www-data: .

After we've created the repositories we need to set up the authentication for apache.

svn01# htpasswd -c /etc/apache2/dav_svn.passwd your-user-name

Now it is time to restart apache.

svn01# service apache2 restart

Congratulations. Your Subversion Server is now ready. Lets head over to your Workstation and checkout the repositories.

Writing The Tasks

At your workstation you can now checkout the repositories.

wks01# svn co http://svn01/svn/common Common
wks01# svn co http://svn01/svn/service Service
wks01# svn co http://svn01/svn/database
wks01# svn co http://svn01/svn/website

First we will add a common task to set up NTP. Here you can add other common tasks later. So change to the Common directory and create a file named NTP.pm.

wks01# cd Common

# Common/NTP.pm
package Common::NTP;
use Rex -base;
task prepare => sub {
   install "ntp";
   file "/etc/ntp.conf",
      source => "files/ntp.conf",
      on_change => sub {
         service ntp => "restart";
      };
};
1;

This creates a task called "prepare". This task gets registered in the "Namespace" NTP. The task will install the package "ntp" if it is not already installed and upload the configuration file to the server. If the content of the file changes it will restart the ntp service. Now, you need to create the ntp.conf file.

bash Common# mkdir files

Paste this to the file files/ntp.conf. This is a simple default ntp.conf file. Of course you can change this to suite your needs :)

# /etc/ntp.conf, managed with rex
driftfile /var/lib/ntp/ntp.drift
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
server 0.ubuntu.pool.ntp.org
server 1.ubuntu.pool.ntp.org
server 2.ubuntu.pool.ntp.org
server 3.ubuntu.pool.ntp.org
# Use Ubuntu's ntp server as a fallback.
server ntp.ubuntu.com
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery
# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1

For the beginning this is enough. So now you can add the new files to the repository.

wks01 Common# svn add NTP.pm files
wks01 Common# svn ci -m "added NTP task"

Now we will add something to the service repository.

wks01 Common# cd ../Service
wks01 Service# touch Apache.pm MySQL.pm

Now paste the following code to the Apache.pm Module.

package Service::Apache;
use Rex -base;
task prepare => sub {
   install "apache2";
};
task configure => sub {
   my $param = shift;
   file "/etc/apache2/apache2.conf",
      owner => "root",
      mode => 644,
      content => template("templates/apache2/apache2.conf.tpl", %{ $param });
   file "/etc/apache2/conf.d/security",
      owner => "root",
      mode => 644,
      content => template("templates/apache2/conf.d/security.tpl", %{ $param });
};
1;

And create the templates used by this module.

wks01 Service# mkdir -p templates/apache2/conf.d

The contents of templates/apache2/apache2.conf.tpl. I've stripped all comments.

LockFile /var/run/apache2/accept.lock
PidFile /var/run/apache2.pid
Timeout <%= is_defined($::timeout, "300") %>
KeepAlive <%= is_defined($::keepalive, "On") %>
MaxKeepAliveRequests <%= is_defined($::max_keepalive_requests, "100") %>
KeepAliveTimeout <%= is_defined($::keepalive_timeout, "5") %>

    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0


    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          150
    MaxRequestsPerChild   0


    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          150
    MaxRequestsPerChild   0

User <%= is_defined($::user, "www-data") %>
Group <%= is_defined($::group, "www-data") %>

AccessFileName .htaccess

    Order allow,deny
    Deny from all
    Satisfy all

DefaultType None

HostnameLookups <%= is_defined($::hostname_lookups, "Off") %>
ErrorLog <%= is_defined($::error_log, "/var/log/apache2/error.log") %>
LogLevel <%= is_defined($::log_level, "warn") %>
Include mods-enabled/*.load
Include mods-enabled/*.conf
Include httpd.conf
Include ports.conf
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

Include conf.d/
Include sites-enabled/

The content of templates/apache2/conf.d/security.tpl. I've stripped all comments.

ServerTokens <%= is_defined($::server_tokens, "Prod") %>
ServerSignature <%= is_defined($::server_signature, "Off") %>
TraceEnable <%= is_defined($::trace_enable, "Off") %>

Now we will continue with the MySQL module. Open the file MySQL.pm and add the following content to it.

package Service::MySQL;
use Rex -base;
task prepare => sub {
   install "mysql-server";
};
task configure => sub {
   my $param = shift;
   file "/etc/mysql/my.cnf",
      owner => "root",
      mode => 644,
      content => template("templates/mysql/my.cnf.tpl", %{ $param });
};
1;

And create the file templates/mysql/my.cnf.tpl.

[mysqld]
user     = <%= is_defined($::user, "mysql") %>
pid-file = /var/run/mysqld/mysqld.pid
socket      = /var/run/mysqld/mysqld.sock
port     = <%= is_defined($::port, "3306") %>
basedir     = /usr
datadir     = /var/lib/mysql
tmpdir      = /tmp
lc-messages-dir   = /usr/share/mysql
skip-external-locking
bind-address      = <%= $::eth0_ip %>
key_buffer     = <%= is_defined($::key_buffer, "16M") %>
max_allowed_packet   = <%= is_defined($::max_allowed_packet, "16M") %>
thread_stack      = <%= is_defined($::thread_stack, "192K") %>
thread_cache_size       = <%= is_defined($::thread_cache_size, "8") %>
myisam-recover         = BACKUP
query_cache_limit = <%= is_defined($::query_cache_limit, "1M") %>
query_cache_size        = <%= is_defined($::query_cache_size, "16M") %>
expire_logs_days  = <%= is_defined($::expire_logs_days, "10") %>
max_binlog_size         = <%= is_defined($::max_binlog_size, "100M") %>

[mysqldump]
quick
quote-names
max_allowed_packet   = <%= is_defined($::max_allowed_packet, "16M") %>
[mysql]
[isamchk]
key_buffer     = <%= is_defined($::key_buffer, "16M") %>
!includedir /etc/mysql/conf.d/

Now add all the files to the repository.

wks01 Service# svn add *
wks01 Service# svn ci -m "inital commit of apache and mysql service"

Okay, now you have your first common modules. Now it is time to create the tasks for the database project.


Change to the database repository.

wks01 Service# cd ../database
wks01 database# touch Rexfile

We will now add our 2 common repositories as an external repository.

wks01 database# svn propedit svn:externals .

Add the following lines into the editor, save and close it.

lib/Common  http://svn01/svn/common
lib/Service http://svn01/svn/service

After you've set the externals, run an update and commit the new settings.

wks01 database# svn up
wks01 database# svn ci -m "added external repositories"

The svn up command now checks both external repositories for updates. So if anyone adds new services to one of this repositories you will get the new services afert an svn up. Now you can start creating your Rexfile to set up a database server.

set user => "root";
set password => "f00b4r";
include qw/
   Common::NTP
   Service::MySQL
/;
set group => srvdb => "db01";
task "prepare", group => "srvdb", sub {
   
   # run the common ntp task "prepare"
   Common::NTP::prepare();
   # install mysql
   Service::MySQL::prepare();
   # configure mysql
   Service::MySQL::configure({
      key_buffer         => "32M",
      max_allowed_packet => "32M",
   });
   # restart mysql
   service mysql => "restart";
};

First (in line 1 and 2) we set the authentication to use against the target host (db01). Here you can also use key authentication (see http://rexify.org/howtos/start.html or more information). Then we include the Common::NTP and Service::MySQL module so that we can use the tasks from these modules. Every task gets registered as a perl function, too. So we can just call them like any other perl functions. After we've included the wanted modules, we define a new server group called "srvdb" and add the host "db01" to it. It is possible to add multiple hosts to a group. For example:

set group => srvdb => "db01", "db02", "db03";
set group => srvdb => "db[01..03]";

In line 12 we define our first task named "prepare". This task gets executed on all servers in the group "srvdb". This task calls the "prepare" task of our Common::NTP and Service::MySQL module. After the setup is done, we configure the mysql server in line 20 and restart the mysql service in line 26. That's all. This will install, configure and start the MySQL Server. You can now commit everything back to the repository.

wks01 database# svn add Rexfile
wks01 database# svn ci -m "inital commit of Rexfile"

To set up your database server you can type the following command:

wks01 database# rex prepare

If you want to list the tasks inside a Rexfile you can use the following command.

wks01 database# rex -T

After our database is running we can now create the tasks for the website project.

Creating The Website Tasks

First, change to the website project directory.

wks01 database# cd ../website
wks01 website# touch Rexfile

First we need to define the externals here as well.

wks01 website# svn propedit svn:externals .

Add the following lines into the editor, save and close it.

lib/Common  http://svn01/svn/common
lib/Service http://svn01/svn/service

After you've set the externals, run an update and commit the new settings.

wks01 database# svn up
wks01 database# svn ci -m "added external repositories"

The svn up command now checks both external repositories for updates. So if anyone adds new services to one of this repositories you will get the new services afert an svn up. Now you can start creating your Rexfile to set up a web server.

# Rexfile
set user => "root";
set password => "test";
include qw/
   Common::NTP
   Service::Apache
/;
set group => srvweb => "web01";
task "prepare", group => "srvweb", sub {
   
   # run the common ntp task "prepare"
   Common::NTP::prepare();
   # install apache
   Service::Apache::prepare();
   # configure apache
   Service::Apache::configure({
      timeout => 60,
   });
   # restart apache
   service apache2 => "restart";
};

Save the file and at it to the repository.

wks01 website# svn add Rexfile
wks01 website# svn ci -m "initial Rexfile"

Now you're ready to set up your webserver with the following command.

wks01 website# rex prepare

After you've set up your database and your webserver you are ready to deploy an application. For example, if you're getting a ZIP or TAR archive from the development team, you can do this by adding a second task to your Rexfile.

task "deploy", group => "srvweb", sub {
   upload "your-web-application.tar.gz", "/tmp";
   extract "/tmp/your-web-application.tar.gz",
      owner => "www-data",
      group => "www-data",
      to    => "/var/www";
   rm "/tmp/your-web-application.tar.gz";
};