Wednesday, November 30, 2011

The Lustre Distributed Filesystem

There comes a time in a network or storage administrator's career
when a large collection of storage volumes needs to be pooled together
and distributed within a clustered or multiple client network, while
maintaining high performance with little to no bottlenecks when accessing
the same files. That is where Lustre comes into the picture. The Lustre
filesystem is a high-performance distributed filesystem intended for
larger network and high-availability environments.

The Storage Area Network and Linux

Traditionally, Lustre is configured to manage remote data storage disk
devices within a Storage Area Network (SAN), which is two or more
remotely attached disk devices communicating via a Small Computer System
Interface (SCSI) protocol. This includes Fibre Channel, Fibre Channel
over Ethernet (FCoE), Serial Attached SCSI (SAS) and even iSCSI. To better
explain what a SAN is, it may be more beneficial to begin with what it
isn't. For instance, a SAN shouldn't be confused with a Local Area Network
(LAN), even if that LAN carries storage traffic (that is, via networked filesystem shares and so on). Only if the LAN carries storage traffic using
the iSCSI or FCoE protocols can it then be considered a SAN. Another
thing that a SAN isn't is Network Attached Storage (NAS). Again, the
SAN relies heavily on a SCSI protocol, while the NAS uses the NFS and
SMB/CIFS file-sharing protocols.
An external storage target device will represent storage volumes as
Logical Units within the SAN. Typically, a set of Logical Units will be
mapped across a SAN to an initiator node—in our case, it would be the
server(s) managing the Lustre filesystem. In turn, the server(s) will
identify one or more SCSI disk devices within its SCSI subsystem and treat
them as if they were local drives. The amount of SCSI disks identified
is determined by the amount of Logical Units mapped to the initiator. If
you want to follow along with the examples here, it is relatively simple
to configure a couple virtual machines: one as the server node with
one or more additional disk devices to export and the second to act as
a client node and mount the Lustre enabled volume. Although it is bad
practice, for testing purposes, it also is possible to have a single
virtual machine configured as both server and client.


SCSI is an ANSI-standardized hardware and software computing
interface adopted by all early storage manufacturers. Revised editions
of the standard continue to be used today.

The Distributed Filesystem

A distributed filesystem allows access to files from multiple hosts
sharing a computer network. This makes it possible for multiple
users on multiple client nodes to share files and storage resources. The
client nodes do not have direct access to the underlying block storage
but interact over the network using a protocol and, thus, make it possible
to restrict access to the filesystem depending on access lists or
capabilities on both the servers and the clients, unlike a clustered filesystem, where all nodes have equal access to the block storage where the
filesystem is located. On these systems, the access control must reside on
the client. Other advantages to utilizing distributed filesystems include
the fact that they may involve facilities for transparent replication
and fault tolerance. So, when a limited number of nodes in a filesystem
goes off-line, the system continues to work without any data loss.
Lustre (or Linux Cluster) is one such distributed filesystem,
usually deployed for large-scale cluster computing. Licensed under
the GNU General Public License (or GPL), Lustre provides a solution in
which high performance and scalability to tens of thousands of nodes
and petabytes of storage becomes a reality and is relatively simple to
deploy and configure. Despite the fact that Lustre 2.0 has been released,
for this article, I work with the generally available 1.8.5.
Lustre contains a somewhat unique architecture, with three major
functional units. One is a single metadata server or MDS that contains
a single metadata target or MDT for each Lustre filesystem. This
stores namespace metadata, which includes filenames, directories, access
permissions and file layout. The MDT data is stored in a single disk filesystem mapped locally to the serving node and is a dedicated filesystem
that controls file access and informs the client node(s) which object(s)
make up a file. Second are one or more object storage servers (OSSes) that
store file data on one or more object storage targets or OST. An OST is a
dedicated object-base filesystem exported for read/write operations. The
capacity of a Lustre filesystem is determined by the sum of the total
capacities of the OSTs. Finally, there's the client(s) that accesses and
uses the file data.
Lustre presents all clients with a unified namespace
for all of the files and data in the filesystem that allow concurrent
and coherent read and write access to the files in the filesystem. When
a client accesses a file, it completes a filename lookup on the MDS,
and either a new file is created or the layout of an existing file is
returned to the client. Locking the file on the OST, the client will
then run one or more read or write operations to the file but will not
directly modify the objects on the OST. Instead, it will delegate tasks to
the OSS. This approach will ensure scalability and improved security and
reliability, as it does not allow direct access to the underlying storage,
thus, increasing the risk of filesystem corruption from misbehaving/defective clients. Although all three components (MDT, OST and client)
can run on the same node, they typically are configured on separate
nodes communicating over a network (see the details on LNET later in this
article). In
this example, I'm running the MDT and OST on a single server node
while the client will be accessing the OST from a separate node.

Installing Lustre

To obtain Lustre 1.8.5, download the prebuilt binaries packaged in RPMs,
or download the source and build the modules and utilities for your
respective Linux distribution. Oracle provides server RPM packages
for both Oracle Enterprise Linux (OEL) 5 and Red Hat Enterprise Linux
(RHEL) 5, while also providing client RPM packages for OEL 5, RHEL 5
and SUSE Linux Enterprise Server (SLES) 10,11. If you will be building
Lustre from source, ensure that you are using a Linux
kernel 2.6.16 or greater. Note that in all deployments of Lustre, the
server that runs on an MDS, MGS (discussed below) or OSS must utilize a
patched kernel. Running a patched kernel on a Lustre client is optional
and required only if the client will be used for multiple purposes,
such as running as both a client and an OST.
If you already have a supported operating system,
make sure that the patched kernel, lustre-modules, lustre-ldiskfs (a
Lustre-patched backing filesystem kernel module package for the ext3 filesystem), lustre (which includes userspace utilities to configure and run
Lustre) and e2fsprogs packages are installed on the host system while
also resolving its dependencies from a local or remote repository. Use
the rpm command to install all necessary packages:

$ sudo rpm -ivh kernel-2.6.18-
$ sudo rpm -ivh lustre-modules-1.8.4-2.6.18_194.
$ sudo rpm -ivh lustre-ldiskfs-3.1.3-2.6.18_194.
$ sudo rpm -ivh lustre-1.8.4-2.6.18_194.

$ sudo rpm -ivh e2fsprogs-1.41.10.sun2-0redhat.oel5.i386.rpm
After these packages have been installed, list the boot directory to
reveal the newly installed patched Linux kernel:

[petros@lustre-host ~]$ ls /boot/
View the /boot/grub/grub.conf file to validate that the newly installed kernel has been set as the default kernel. Now that all packages have been installed, a reboot is required to load the new kernel image. Once the system has been rebooted, an invocation of the uname command will reveal the currently booted kernel image:

[petros@lustre-host ~]$ uname -a
Linux lustre-host 2.6.18- #1 
 ↪SMP Mon Jul 26 22:12:56 MDT 2010 i686 i686 i386 GNU/Linux
Meanwhile, on the client side, the packages for the lustre client (utilities for patchless clients) and lustre client modules (modules for patchless clients) need to be installed on all desired client machines:

$ sudo rpm -ivh lustre-client-1.8.4-2.6.18_194.
$ sudo rpm -ivh lustre-client-modules-1.8.4-2.6.18_194.
Note that these client machines need to be within the same network as the host machine serving the Lustre filesystem. After the packages are installed, reboot all affected client machines.

Configuring Lustre Server

In order to configure the Lustre filesystem, you need to configure Lustre Networking, or LNET, which provides the communication infrastructure required by the Lustre filesystem. LNET supports many commonly used network types, which include InfiniBand and IP networks. It allows simultaneous availability across multiple network types with routing between them. In this example, let's use tcp, so use your favorite editor to append the following line to the /etc/modprobe.conf file:

options lnet networks=tcp
This step restricts LNET to be using only the specified network interfaces and prevents LNET from using all network interfaces.
Before moving on, it is important to discuss the role of the Management Server or MGS. The MGS stores configuration information for all Lustre filesystems in a clustered setup. An OST contacts the MGS to provide information, while the client(s) contact the MGS to retrieve information. The MGS requires its own disk for storage, although there is a provision that allows the MGS to share a disk with a single MDT. Type the following command to create a combined MGS/MDT node:

[petros@lustre-host ~]$ sudo /usr/sbin/mkfs.lustre 
 ↪--fsname=lustre --mgs --mdt /dev/sda1

   Permanent disk data:
Target:     lustre-MDTffff
Index:      unassigned
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x75
              (MDT MGS needs_index first_time update )
Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro
Parameters: mdt.group_upcall=/usr/sbin/l_getgroups

device size = 1019MB
2 6 18
formatting backing filesystem ldiskfs on /dev/sda1
        target name  lustre-MDTffff
        4k blocks     261048
        options        -i 4096 -I 512 -q -O dir_index,uninit_groups -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre-MDTffff  -i 4096 -I 512 -q -O 
 ↪dir_index,uninit_groups -F /dev/sda1 261048
Writing CONFIGS/mountdata
If nothing has been provided, by default, the fsname is lustre. If one or more of these filesystems are created, it is necessary to use unique names for each labeled volume. These names become very important for when you access the target on the client system.
Create the OST by executing the following command:

[petros@lustre-host ~]$ sudo /usr/sbin/mkfs.lustre --ost 
 ↪--fsname=lustre --mgsnode= /dev/sda2

   Permanent disk data:
Target:     lustre-OSTffff
Index:      unassigned
Lustre FS:  lustre
Mount type: ldiskfs
Flags:      0x72
              (OST needs_index first_time update )
Persistent mount opts: errors=remount-ro,extents,mballoc
Parameters: mgsnode=

checking for existing Lustre data: not found
device size = 1027MB
2 6 18
formatting backing filesystem ldiskfs on /dev/sda2
        target name  lustre-OSTffff
        4k blocks     263064
        options        -J size=40 -i 16384 -I 256 -q -O 
 ↪dir_index,extents,uninit_groups -F
mkfs_cmd = mke2fs -j -b 4096 -L lustre-OSTffff  
 ↪-J size=40 -i 16384 -I 256 -q -O 
 ↪dir_index,extents,uninit_groups -F /dev/sda2 263064
Writing CONFIGS/mountdata
When the target needs to provide information to the MGS or when the client accesses the target for information lookup, all also will need to be aware of where the MGS is, which you have defined for this target as the server's IP address followed by the network interface for communication. This is just a reminder that the interface was defined earlier in the /etc/modprobe.conf file.
Now you easily can mount these newly formatted devices local to the host machine. Mount the MDT:

[petros@lustre-host ~]$ sudo mkdir -p /mnt /mdt
[petros@lustre-host ~]$ sudo mount -t lustre /dev/sda1 /mnt/mdt/
Verify that it is mounted with the df command:

[petros@lustre-host ~]$ df -t lustre
Filesystem          1K-blocks   Used  Available Use% Mounted on
/dev/sda1              913560   17752 843600    3%   /mnt/mdt
The /var/log/messages file will log the following messages:

Jun 25 10:26:54 lustre-host kernel: Lustre: lustre-MDT0000: 
 ↪new disk, initializing
Jun 25 10:26:54 lustre-host kernel: Lustre: lustre-MDT0000: 
 ↪Now serving lustre-MDT0000 on /dev/sda1 with recovery enabled
Jun 25 10:26:54 lustre-host kernel: Lustre: 
 ↪lustre-MDT0000: group upcall set to /usr/sbin/l_getgroups
Jun 25 10:26:54 lustre-host kernel: Lustre: lustre-MDT0000.mdt: 
 ↪set parameter group_upcall=/usr/sbin/l_getgroups
Mount the OST:

[petros@lustre-host ~]$ sudo mkdir -p /mnt /ost
[petros@lustre-host ~]$ sudo mount -t lustre /dev/sda2 /mnt/ost/
Verify that it is mounted with the df command:

[petros@lustre-host ~]$ df -t lustre
Filesystem        1K-blocks      Used Available Use% Mounted on
/dev/sda1            913560     17768    843584   3% /mnt/mdt
/dev/sda2           1035692     42460    940620   5% /mnt/ost
The /var/log/messages file will log the following messages:

Jun 25 10:41:33 lustre-host kernel: Lustre: 
 ↪lustre-OST0000: new disk, initializing
Jun 25 10:41:33 lustre-host kernel: Lustre: 
 ↪lustre-OST0000: Now serving lustre-OST0000 on 
 ↪/dev/sda2 with recovery enabled
Jun 25 10:41:39 lustre-host kernel: Lustre: 
 ↪3417:0:(mds_lov.c:1155:mds_notify()) MDS lustre-MDT0000: 
 ↪add target lustre-OST0000_UUID
Jun 25 10:41:39 lustre-host kernel: Lustre: 
 ↪Only 0/1 OSTs are active, abort quota recovery
Jun 25 10:41:39 lustre-host kernel: Lustre: lustre-OST0000: 
 ↪received MDS connection from 0@lo
Jun 25 10:41:39 lustre-host kernel: Lustre: MDS lustre-MDT0000: 
 ↪lustre-OST0000_UUID now active, resetting orphans
Even though they are mounted like typical filesystems, you will notice that the MDT and OST labeled volumes do not contain the typical filesystem characteristics. That is where the client will come into play:

[petros@lustre-host ~]$ ls /mnt/ost/
ls: /mnt/ost/: Not a directory
[petros@lustre-host ~]$ sudo ls /mnt/ost/
ls: /mnt/ost/: Not a directory
[petros@lustre-host ~]$ sudo ls -l /mnt/
total 4
d--------- 1 root root    0 Jun 25 10:22 ost
d--------- 1 root root    0 Jun 25 10:22 mdt

Configuring Lustre Client(s)

Remember that you named the Lustre-enabled volume lustre. When you mount the volume over the network on the client node, you must specify this name. This can be observed below following the network method (tcp0) by which you are mounting the remote volume. Again, TCP was defined in the /etc/modprobe.conf file for the supported LNET networks interfaces:

[petros@client1 ~]$ sudo mkdir -p /lustre
[petros@client1 ~]$ sudo mount -t lustre 
 ↪ /lustre/
After successfully mounting the remote volume, you will see the /var/log/messages file append:

Jun 25 10:44:17 client1 kernel: Lustre: 
 ↪Client lustre-client has started
Use df to list the mounted Lustre-enabled volumes:

[petros@client1 ~]$ df -t lustre
Filesystem        1K-blocks Used  Available Use% Mounted on 1035692   42460 940556    5%   /lustre
Once mounted, the filesystem now can be accessed by the client node. For instance, you cannot read or write from or to files located on the mounted OST. In the following example, let's write approximately 40MB of data to a new file, on /lustre:

[petros@client1 lustre]$ sudo dd if=/dev/zero 
 ↪of=/lustre/test.dat bs=4M count=10
10+0 records in
10+0 records out
41943040 bytes (42 MB) copied, 1.30103 seconds, 32.2 MB/s
A df listing will reflect the changes of available and used capacities to the /lustre mountpoint:

[petros@client1 lustre]$ df -t lustre
Filesystem              1K-blocks   Used  Available Use% Mounted on  1035692     83420 899596    9%   /lustre
[petros@client1 lustre]$ ls -l
total 40960
-rw-r--r-- 1 root root 41943040 Jun 25 10:47 test.dat


Although I covered a simple introduction and tutorial of the Lustre distributed filesystem, there still is so much uncharted territory and methods by which the filesystem can be configured for a truly rich computing environment. For instance, Lustre can be configured for high availability to ensure that in the situation of failure, the system's services continue without interruption. That is, the accessibility between the client(s), server(s) and external target storage is always made available through a process called failover. Additional HA is provided through an implementation of disk drive redundancy in some sort of RAID (hardware or software via mdadm) configuration for the event of drive failures. These high-availability techniques normally would apply to server nodes hosting the MDS and OSS. It is suggested to place the OST storage on a RAID 5 or, preferably, RAID 6, while the MDT storage should be RAID 1 or RAID 0+1.
It isn't a difficult technology to work with; however, there still exists an entire community with excellent administrator and developer resources ranging from articles, mailing lists and more to aid the novice in installing and maintaining a Lustre-enabled system. Commercial support for Lustre is made available by a non-exhaustive list of vendors selling bundled computing and Lustre storage systems. Many of these same vendors also are contributing to the Open Source community surrounding the Lustre Project.


Failover is a process with the capability to switch over automatically to a redundant or standby computer server, system or network upon the failure or abnormal termination of the previously active application, server, system or network. Many failover solutions exist on the Linux platform to cover both SAN and LAN environments.


Lustre Project Page:
Wikipedia: Distributed File Systems:

Tuesday, November 29, 2011

Dnsmasq For Easy LAN Name Services

When you want a good reliable and easy-to-configure LAN name server, try Dnsmasq. Dnsmasq does DHCP, DNS, DNS caching, and TFTP, so it's four servers in one. If you have no public servers it should meet all of your needs, and it's a great complement to an authoritative name server. In this tutorial we'll learn how to deliver all network configurations to our LAN hosts through DHCP.


All of your network hosts must have their own hostnames, and be configured to get their network configurations via DHCP (dynamic host configuration protocol.) If you have some machines with static IP addresses Dnsmasq can incorporate them as well, so there is no need to change them. You should have a correctly-configured network, and all hosts able to ping each other.

A Bit of Terminology

Let's review some basic terms so we know what the heck we're talking about.
An authoritative name server is for publishing the addresses of public servers. If you have an Internet-facing server such as a Web site, mail server, or FTP server, then somewhere there is an authoritative server that advertises their IP addresses and names. This may be an authoritative DNS (domain name services) server on your premises, or managed by a third party like your Internet service provider or a hosting service. You can query any public server with the dig command to see how its name and IP address are matched up:

$ dig +nocmd +noall +answer  5276 IN  A  5276 IN  A

Think of an authoritative DNS server as the master address book for an Internet domain. This address book is copied to the world's root DNS servers, and then copied by countless other servers all over the Internet. It is a beautiful distributed system that provides speed and fault-tolerance. Keeping authoritative servers completely separate from the other types of name servers — recursive and caching — is a fundamental security practice. So you might use BIND, PowerDNS, or MaraDNS for your authoritative server, and Dnsmasq for private LAN name services and caching.
dns cache is a local copy of the addresses of sites you have visited. This speeds up your network performance because network applications don't have to wait for DNS queries to be answered by remote servers.
recursive name server is the one that looks up the address of sites you want to visit. Recursive and cache functions are often combined in the same server. For example, when you configure the DNS for your Internet account, your ISP's DNS servers are most likely recursive and caching servers. Public DNS servers like Google Public DNS and OpenDNS are recursive and caching servers. Sometimes you get can speed up your Internetworking by using different third-party servers; try Namebench to help you find the fastest ones. Dnsmasq is not a recursive name server, but it can be configured to query any recursive server you want.
Trivial File Transfer Protocol (TFTP) is a very simple, insecure FTP server used inside private networks for network booting of PCs and embedded devices like routers and VoIP (voice over IP) endpoints.

Global Settings

Dnsmasq is configured in /etc/dnsmasq.conf. I recommend copying the original to keep as a reference, and start over with a blank file. Every time you make a change to dnsmasq.conf you have to restart Dnsmasq. In these here modern times there are multiple ways to do this, hurrah, though running/etc/init.d/dnsmasq restart still works on most distros.
For this article let's assume a small network with two subnets: one wired and one wireless, at and Dnsmasq is installed on a LAN router with both wired and wireless interfaces at and First let's take care of some important global settings:


Adding domain-needed blocks incomplete requests from leaving your network, such as google instead of google.combogus-priv prevents non-routable private addresses from being forwarded out of your network. Using these is simply good netizenship.
Set your private domain name with, replacing mydomain with any domain name your heart desires. You don't need to register it with a domain name registrar because it's private and never leaves your LAN.
The expand-hosts directive adds the domain name to your hostnames, so you get fully-qualified domain names like Again, these are completely arbitrary and can be whatever you want.
local=/ ensures that queries for your private domain are only answered by Dnsmasq, from /etc/hosts or DHCP.
The listen-address directive tells Dnsmasq which interface or interfaces to listen on. Always uselisten-address because you don't want Dnsmasq exposed to the wrong networks, and especially not the Internet. Always include the loopback address. You could use the interface= directive instead, for example interface=eth0, but the Linux kernel doesn't always bring up network interfaces with the same names after reboot. If you have more than one NIC the names could get changed, and then your name services will be messed up.
The bind-interfaces directive ensures that Dnsmasq will listen only to the addresses specificied withlisten-address.

Configuring DHCP

Now let's set up DHCP for our two subnets. This is so easy you will dance for joy:


I like to reserve addresses below .100 for servers. This example supplies a hundred DHCP addresses per subnet. Note that they are labeled with the tags lan and wifi. This is a brilliantly simple system that simplifies delivering different services to different subnets, as in the following examples:

#set default gateway

#set DNS server

The first stanza sets the default route for each subnet. The number 3 tag means router. You can see all the tag numbers with the dnsmasq --help dhcp command. The second stanza tells our LAN clients to get their DNS from the Dnsmasq server.

Upstream Name Servers

You need to tell Dnsmasq where to forward Internet DNS requests. This could be your ISP's nameservers, or any DNS service you want to use. It is good to use at least two completely different services. This example uses Google Public DNS and OpenDNS:


Static IP Addresses

Dnsmasq painlessly incorporates hosts with static IP addresses into your local DNS. Suppose you have three servers with static addresses; all you do is add them to the /etc/hosts file on the Dnsmasq server:      localhost   server1   server2   server3

Always include the localhost line.

TFTP Server

You can enable Dnsmasq's built-in TFTP server by adding this line to dnsmasq.conf:
And then you'll need to set up your boot directory and pxelinux configuration, which is a subject for another day. If you already have a working TFTP/pxelinux server, then point Dnsmasq to it like this, using your own server name and address:
Once again we have run out of paper and it is time to end. Please visit Dnsmasq to learn more about this excellent server.

Linux: 25 PHP Security Best Practices For Sys Admins

PHP is an open-source server-side scripting language and it is a widely used. The Apache web server provides access to files and content via the HTTP OR HTTPS protocol. A misconfigured server-side scripting language can create all sorts of problems. So, PHP should be used with caution. Here are twenty-fivephp security best practices for sysadmins for configuring PHP securely.

Our Sample Setup For PHP Security Tips

  • DocumentRoot: /var/www/html
  • Default Web server: Apache ( you can use Lighttpd or Nginx instead of Apache)
  • Default PHP configuration file: /etc/php.ini
  • Default PHP extensions config directory: /etc/php.d/
  • Our sample php security config file: /etc/php.d/security.ini (you need to create this file using a text editor)
  • Operating systems: RHEL / CentOS / Fedora Linux (the instructions should work with any other Linux distributions such as Debian / Ubuntu or other Unix like operating systems such as OpenBSD/FreeBSD/HP-UX).
  • Default php server TCP/UDP ports: none
Most of the actions listed in this post are written with the assumption that they will be executed by the root user running the bash or any other modern shell:
$ php -v
Sample outputs:
PHP 5.3.3 (cli) (built: Oct 24 2011 08:35:41)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies
For demonstration purpose I'm going to use the following operating system:
$ cat /etc/redhat-release
Sample outputs:
Red Hat Enterprise Linux Server release 6.1 (Santiago)

#1: Know Your Enemy

PHP based apps can face the different types of attacks. I have noticed the different types of attacks:
  1. XSS - Cross-site scripting is a vulnerability in php web applications, which attackers may exploit to steal users' information. You can configure Apache and write more secure PHP scripts (validating all user input) to avoid xss attacks.
  2. SQL injection - It is a vulnerability in the database layer of an php application. When user input is incorrectly filtered any SQL statements can be executed by the application. You can configure Apache and write secure code (validating and escaping all user input) to avoid SQL injection attacks. A common practice in PHP is to escape parameters using the function called mysql_real_escape_string() before sending the SQL query.
  3. File uploads - It allows your visitor to place files (upload files) on your server. This can result into various security problems such as delete your files, delete database, get user details and much more. You can disable file uploads using php or write secure code (like validating user input and only allow image file type such as png or gif).
  4. Including local and remote files - An attacker can open files from remote server and execute any PHP code. This allows them to upload file, delete file and install backdoors. You can configure php to disable remote file execution.
  5. eval() - Evaluate a string as PHP code. This is often used by an attacker to hide their code and tools on the server itself. You can configure php to disable eval().
  6. Sea-surf Attack (Cross-site request forgery - CSRF) - This attack forces an end user to execute unwanted actions on a web application in which he/she is currently authenticated. A successful CSRF exploit can compromise end user data and operation in case of normal user. If the targeted end user is the administrator account, this can compromise the entire web application.

#2: Find Built-in PHP Modules

To see the set of compiled-in PHP modules type the following command:
# php -m
Sample outputs:
[PHP Modules]
[Zend Modules]
I recommends that you use PHP with a reduced modules for performance and security. For example, you can disable sqlite3 module by deleting (removing) configuration file , ORrenaming (moving) a file called /etc/php.d/sqlite3.ini as follows:
rm /etc/php.d/sqlite3.ini
mv /etc/php.d/sqlite3.ini /etc/php.d/sqlite3.disable
Other compiled-in modules can only be removed by reinstallating PHP with a reduced configuration. You can download php source code from and compile it as follows with GD, fastcgi, and MySQL support:
./configure --with-libdir=lib64 --with-gd --with-mysql --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/usr/com --mandir=/usr/share/man --infodir=/usr/share/info --cache-file=../config.cache --with-config-file-path=/etc --with-config-file-scan-dir=/etc/php.d  --enable-fastcgi --enable-force-cgi-redirect

#3: Restrict PHP Information Leakage

To restrict PHP information leakage disable expose_php. Edit /etc/php.d/secutity.ini and set the following directive:
When enabled, expose_php reports to the world that PHP is installed on the server, which includes the PHP version within the HTTP header (e.g., X-Powered-By: PHP/5.3.3). The PHP logo guids (see example) are also exposed, thus appending them to the URL of a PHP enabled site will display the appropriate logo. When expose_php enabled you can see php version using the following command:
$ curl -I
Sample outputs:
HTTP/1.1 200 OK
X-Powered-By: PHP/5.3.3
Content-type: text/html; charset=UTF-8
Vary: Accept-Encoding, Cookie
X-Vary-Options: Accept-Encoding;list-contains=gzip,Cookie;string-contains=wikiToken;string-contains=wikiLoggedOut;string-contains=wiki_session
Last-Modified: Thu, 03 Nov 2011 22:32:55 GMT
I also recommend that you setup the ServerTokens and ServerSignature directives in httpd.conf to hide Apache version and other information.

#4: Minimize Loadable PHP Modules (Dynamic Extensions)

PHP supports "Dynamic Extensions". By default, RHEL loads all the extension modules found in /etc/php.d/ directory. To enable or disable a particular module, just find the configuration file in /etc/php.d/ directory and comment the module name. You can also rename or delete module configuration file. For best PHP performance and security, you should only enable the extensions your webapps requires. For example, to disable gd extension, type the following commands:
# cd /etc/php.d/
# mv gd.{ini,disable}
/sbin/service httpd restart

To enable php module called gd, enter:
# mv gd.{disable,ini}
/sbin/service httpd restart

#5: Log All PHP Errors

Do not expose PHP error messages to all site visitors. Edit /etc/php.d/security.ini and set the following directive:

#6: Disallow Uploading Files

Edit /etc/php.d/security.ini and set the following directive to disable file uploads for security reasons:
If users of your application need to upload files, turn this feature on by settingupload_max_filesize limits the maximum size of files that PHP will accept through uploads:
# user can only upload upto 1MB via php

#7: Turn Off Remote Code Execution

If enabled, allow_url_fopen allows PHP's file functions -- such as file_get_contents() and the include and require statements -- can retrieve data from remote locations, like an FTP or web site.
The allow_url_fopen option allows PHP's file functions - such as file_get_contents() and the include and require statements - can retrieve data from remote locations using ftp or http protocols. Programmers frequently forget this and don't do proper input filtering when passing user-provided data to these functions, opening them up to code injection vulnerabilities. A large number of code injection vulnerabilities reported in PHP-based web applications are caused by the combination of enabling allow_url_fopen and bad input filtering. Edit /etc/php.d/security.ini and set the following directive:
I also recommend to disable allow_url_include for security reasons:

#8: Enable SQL Safe Mode

Edit /etc/php.d/security.ini and set the following directive:
If turned On, mysql_connect() and mysql_pconnect() ignore any arguments passed to them. Please note that you may have to make some changes to your code. Third party and open source application such as WordPress, and others may not work at all when sql.safe_mode enabled. I also recommend that you turn off magic_quotes_gpc for all php 5.3.x installations as the filtering by it is ineffective and not very robust. mysql_escape_string() and custom filtering functions serve a better purpose (hat tip to Eric Hansen):

#9: Control POST Size

The HTTP POST request method is used when the client (browser or user) needs to send data to the Apache web server as part of the request, such as when uploading a file or submitting a completed form. Attackers may attempt to send oversized POST requests to eat your system resources. You can limit the maximum size POST request that PHP will process. Edit /etc/php.d/security.ini and set the following directive:
; Set a realistic value here
The 1K sets max size of post data allowed by php apps. This setting also affects file upload. To upload large files, this value must be larger than upload_max_filesize. I also suggest that you limit available methods using Apache web server. Edit, httpd.conf and set the following directive for DocumentRoot /var/www/html:

        Order allow,deny
## Add rest of the config goes here... ##


#10: Resource Control (DoS Control)

You can set maximum execution time of each php script, in seconds. Another recommend option is to set maximum amount of time each script may spend parsing request data, and maximum amount of memory a script may consume. Edit /etc/php.d/security.ini and set the following directives:
# set in seconds
max_execution_time =  30
max_input_time = 30
memory_limit = 40M

#11: Install Suhosin Advanced Protection System for PHP

From the project page:
Suhosin is an advanced protection system for PHP installations. It was designed to protect servers and users from known and unknown flaws in PHP applications and the PHP core. Suhosin comes in two independent parts, that can be used separately or in combination. The first part is a small patch against the PHP core, that implements a few low-level protections against bufferoverflows or format string vulnerabilities and the second part is a powerful PHP extension that implements all the other protections.
See how to install and configure suhosin under Linux operating systems.

#12 Disabling Dangerous PHP Functions

PHP has a lot of functions which can be used to crack your server if not used properly. You can set list of functions in /etc/php.d/security.ini using disable_functions directive:
disable_functions =exec,passthru,shell_exec,system,proc_open,popen,curl_exec,curl_multi_exec,parse_ini_file,show_source

#13 PHP Fastcgi / CGI - cgi.force_redirect Directive

PHP work with FastCGI. Fascgi reduces the memory footprint of your web server, but still gives you the speed and power of the entire PHP language. You can configureApache2+PHP+FastCGI or cgi as described here. The configuration directive cgi.force_redirect prevents anyone from calling PHP directly with a URL like Turn on cgi.force_redirect for security reasons. Edit /etc/php.d/security.ini and set the following directive:
; Enable cgi.force_redirect for security reasons in a typical *Apache+PHP-CGI/FastCGI* setup

#14 PHP User and Group ID

mod_fastcgi is a cgi-module for Apache web server. It can connect to an external FASTCGI server. You need to make sure php run as non-root user. If PHP executes as a root or UID under 100, it may access and/or manipulate system files. You must execute PHP CGIs as a non-privileged user using Apache's suEXEC or mod_suPHP. The suEXEC feature provides Apache users the ability to run CGI programs under user IDs different from the user ID of the calling web server. In this example, my php-cgi is running as phpcgi user and apache is running as apache user:
# ps aux | grep php-cgi
Sample outputs:
phpcgi      6012  0.0  0.4 225036 60140 ?        S    Nov22   0:12 /usr/bin/php-cgi
phpcgi      6054  0.0  0.5 229928 62820 ?        S    Nov22   0:11 /usr/bin/php-cgi
phpcgi      6055  0.1  0.4 224944 53260 ?        S    Nov22   0:18 /usr/bin/php-cgi
phpcgi      6085  0.0  0.4 224680 56948 ?        S    Nov22   0:11 /usr/bin/php-cgi
phpcgi      6103  0.0  0.4 224564 57956 ?        S    Nov22   0:11 /usr/bin/php-cgi
phpcgi      6815  0.4  0.5 228556 61220 ?        S    00:52   0:19 /usr/bin/php-cgi
phpcgi      6821  0.3  0.5 228008 61252 ?        S    00:55   0:12 /usr/bin/php-cgi
phpcgi      6823  0.3  0.4 225536 58536 ?        S    00:57   0:13 /usr/bin/php-cgi
You can use tool such as spawn-fcgi to spawn remote and local FastCGI processes as phpcgi user (first, add phpcgi user to the system):
# spawn-fcgi -a -p 9000 -u phpcgi -g phpcgi -f /usr/bin/php-cgi
Now, you can configure ApacheLighttpd, and Nginx web server to use external php FastCGI running on port 9000 at IP address.

#15 Limit PHP Access To File System

The open_basedir directive set the directories from which PHP is allowed to access files using functions like fopen(), and others. If a file is outside of the paths defined by open_basdir, PHP will refuse to open it. You cannot use a symbolic link as a workaround. For example only allow access to /var/www/html directory and not to /var/www, or /tmp or /etc directories:
; Limits the PHP process from accessing files outside
; of specifically designated directories such as /var/www/html/
; ------------------------------------
; Multiple dirs example
; open_basedir="/home/httpd/vhost/"
; ------------------------------------

#16 Session Path

Session support in PHP consists of a way to preserve certain data across subsequent accesses. This enables you to build more customized applications and increase the appeal of your web site. This path is defined in /etc/php.ini file and all data related to a particular session will be stored in a file in the directory specified by the session.save_path option. The default is as follows under RHEL/CentOS/Fedora Linux:
; Set the temporary directory used for storing files when doing file upload
Make sure path is outside /var/www/html and not readable or writeable by any other system users:
# ls -Z /var/lib/php/
Sample outputs:
drwxrwx---. root apache system_u:object_r:httpd_var_run_t:s0 session
Note: The -Z option to the ls command display SELinux security context such as file mode, user, group, security context and file name.

#17 Keep PHP, Software, And OS Up to Date

Applying security patches is an important part of maintaining Linux, Apache, PHP, and MySQL server. All php security update should be reviewed and applied as soon as possible using any one of the following tool (if you're installing PHP via a package manager):
yum update
apt-get update && apt-get upgrade
You can configure Red hat / CentOS / Fedora Linux to send yum package update notification via email. Another option is to apply all security updates via a cron job. Under Debian / Ubuntu Linux you can use apticron to send security notifications.
Note: Check for the most recent release for source code installations.

#18: Restrict File and Directory Access

Make sure you run Apache as a non-root user such as Apache or www. All files and directory should be owned by non-root user (or apache user) under /var/www/html:
chown -R apache:apache /var/www/html/
/var/www/html/ is a subdirectory and DocumentRoot which is modifiable by other users since root never executes any files out of there, and shouldn't be creating files in there.
Make sure file permissions are set to 0444 (read-only) under /var/www/html/:
# chmod -R 0444 /var/www/html/
Make sure all directories permissions are set to 0445 under /var/www/html/:
find /var/www/html/ -type d -print0 | xargs -0 -I {} chmod 0445 {}

A Note About Setting Up Correct File Permissions

The chown and chmod command make sures that under no circumstances DocumentRoot or files contained in DocumentRoot are writable by the Web server user apache. Please note that you need to set permissions that makes the most sense for the development model of your website, so feel free to adjust the chown and chmod command as per your requirements. In this example, the Apache server run as apache user. This is configured with the User and Groupdirectives in your httpd.conf file. The apache user needs to have read access to everything under DocumentRoot but should not have write access to anything.
Make sure httpd.conf has the following directives for restrictive configuration:

    Options None
    AllowOverride None
    Order allow,deny

You should only grant write access when required. Some web applications such as wordpress and others may need a caching directory. You can grant a write access to caching directory using the following commands:
# chmod a+w /var/www/html/blog/wp-content/cache
### block access to all ###
# echo 'deny from all' > /var/www/html/blog/wp-content/cache/.htaccess

#19: Write Protect Apache, PHP, and, MySQL Configuration Files

Use the chattr command to write protect configuration files:
# chattr +i /etc/php.ini
# chattr +i /etc/php.d/*
# chattr +i /etc/my.ini
# chattr +i /etc/httpd/conf/httpd.conf
# chattr +i /etc/
The chattr command can write protect your php file or files in /var/www/html directory too:
# chattr +i /var/www/html/file1.php
# chattr +i /var/www/html/

#20: Use Linux Security Extensions (such as SELinux)

Linux comes with various security patches which can be used to guard against misconfigured or compromised server programs. If possible use SELinux and other Linux security extensions to enforce limitations on network and other programs. For example, SELinux provides a variety of security policies for Linux kernel and Apache web server. To list all Apache SELinux protection variables, enter:
# getsebool -a | grep httpd
Sample outputs:
allow_httpd_anon_write --> off
allow_httpd_mod_auth_ntlm_winbind --> off
allow_httpd_mod_auth_pam --> off
allow_httpd_sys_script_anon_write --> off
httpd_builtin_scripting --> on
httpd_can_check_spam --> off
httpd_can_network_connect --> off
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_memcache --> off
httpd_can_network_relay --> off
httpd_can_sendmail --> off
httpd_dbus_avahi --> on
httpd_enable_cgi --> on
httpd_enable_ftp_server --> off
httpd_enable_homedirs --> off
httpd_execmem --> off
httpd_read_user_content --> off
httpd_setrlimit --> off
httpd_ssi_exec --> off
httpd_tmp_exec --> off
httpd_tty_comm --> on
httpd_unified --> on
httpd_use_cifs --> off
httpd_use_gpg --> off
httpd_use_nfs --> off
To disable Apache cgi support, enter:
# setsebool -P httpd_enable_cgi off
See Red Hat SELinux guide for more information.

#21 Install Mod_security

ModSecurity is an open source intrusion detection and prevention engine for web applications. You can easily install mod_security under Linux and protect apache and php based apps from xss and various other attacks:
## A few Examples ##
# Do not allow to open files in /etc/
SecFilter /etc/
# Stop SQL injection
SecFilter "delete[[:space:]]+from"
SecFilter "select.+from"

#22 Run Apache / PHP In a Chroot Jail If Possible

Putting PHP and/or Apache in a chroot jail minimizes the damage done by a potential break-in by isolating the web server to a small section of the filesystem. You can use traditional chroot kind of setup with Apache. However, I recommend FreeBSD jailsXEN virtulizationKVM virtulization, or OpenVZ virtualization which uses the concept of containers.

#23 Use Firewall To Restrict Outgoing Connections

The attacker will download file locally on your web-server using tools such as wget. Use iptables to block outgoing connections from apache user. The ipt_owner module attempts to match various characteristics of the packet creator, for locally generated packets. It is only valid in the OUTPUT chain. In this example, allow vivek user to connect outside using port 80 (useful for RHN or centos repo access):
/sbin/iptables -A OUTPUT -o eth0 -m owner --uid-owner vivek -p tcp --dport 80 -m state --state NEW,ESTABLISHED  -j ACCEPT
Here is another example that blocks all outgoing connections from apache user except to our own smtp server, and spam validation API service:
# ....
/sbin/iptables --new-chain apache_user
/sbin/iptables --append OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables --append OUTPUT -m owner --uid-owner apache -j apache_user
# allow apache user to connec to our smtp server
/sbin/iptables --append apache_user -p tcp --syn -d --dport 25 -j RETURN
# Allow apache user to connec to api server for spam validation
/sbin/iptables --append apache_user -p tcp --syn -d --dport 80 -j RETURN
/sbin/iptables --append apache_user -p tcp --syn -d --dport 80 -j RETURN
/sbin/iptables --append apache_user -p tcp --syn -d --dport 80 -j RETURN
/sbin/iptables --append apache_user -p tcp --syn -d --dport 80 -j RETURN
## Add more rules here ##
# No editing below
# Drop everything for apache outgoing connection
/sbin/iptables --append apache_user -j REJECT

#24 Watch Your Logs & Auditing

Check the apache log file:
# tail -f /var/log/httpd/error_log
# grep 'login.php' /var/log/httpd/error_log
# egrep -i "denied|error|warn" /var/log/httpd/error_log

Check the php log file:
# tail -f /var/log/httpd/php_scripts_error.log
# grep "...etc/passwd" /var/log/httpd/php_scripts_error.log

Log files will give you some understanding of what attacks is thrown against the server and allow you to check if the necessary level of security is present or not. The auditd service is provided for system auditing. Turn it on to audit SELinux events, authetication events, file modifications, account modification and so on. I also recommend using standard "Linux System Monitoring Tools" for monitoring your web-server.

#25 Run Service Per System or VM Instance

For large installations it is recommended that you run, database, static, and dynamic content from different servers.
/ ISP/Router /
     | LB01       |
     +------------+                 +--------------------------+
                  |                 | |
                                    | |
(Fig.01: Running Services On Separate Servers)
Run different network services on separate servers or VM instances. This limits the number of other services that can be compromised. For example, if an attacker able to successfully exploit a software such as Apache flow, he / she will get an access to entire server including other services running on the same server (such as MySQL, e-mail server and so on). But, in the above example content are served as follows:
  1. - Use lighttpd or nginx server for static assets such as js/css/images.
  2. and - Apache web-server with php used for generating dynamic content.
  3. - MySQL database server.
  4. - Memcached server is very fast caching system for MySQL. It uses libevent or epoll (Linux runtime) to scale to any number of open connections and uses non-blocking network I/O.
  5. LB01 - A nginx web and reverse proxy server in front of Apache Web servers. All connections coming from the Internet addressed to one of the Web servers are routed through the nginx proxy server, which may either deal with the request itself or pass the request wholly or partially to the main web servers. LB01 provides simple load-balancing.

#26 Additional Tools

From the project page:
PHPIDS (PHP-Intrusion Detection System) is a simple to use, well structured, fast and state-of-the-art security layer for your PHP based web application. The IDS neither strips, sanitizes nor filters any malicious input, it simply recognizes when an attacker tries to break your site and reacts in exactly the way you want it to.
You can use PHPIDS to detect malicious users, and log any attacks detected for later review. Please note that I've personally not used this tool.
From the project page:
PhpSecInfo provides an equivalent to the phpinfo() function that reports security information about the PHP environment, and offers suggestions for improvement. It is not a replacement for secure development techniques, and does not do any kind of code or app auditing, but can be a useful tool in a multilayered security approach.
Security Information About PHP Application
Fig.02: Security Information About PHP Application
See Linux security hardening tips which can reduce available vectors of attack on the system.

A Note About PHP Backdoors

You may come across php scripts or so called common backdoors such as c99, c99madshell, r57 and so on. A backdoor php script is nothing but a hidden script for bypassing all authentication and access your server on demand. It is installed by an attackers to access your server while attempting to remain undetected. Typically a PHP (or any other CGI script) script by mistake allows inclusion of code exploiting vulnerabilities in the web browser. An attacker can use such exploiting vulnerabilities to upload backdoor shells which can give him or her a number of capabilities such as:
  • Download files
  • Upload files
  • Install rootkits
  • Set a spam mail servers / relay server
  • Set a proxy server to hide tracks
  • Take control of server
  • Take control of database server
  • Steal all information
  • Delete all information and database
  • Open TCP / UDP ports and much more

Tip: How Do I Search PHP Backdoors?

Use Unix / Linux grep command to search c99 or r57 shell:
# grep -iR 'c99' /var/www/html/
# grep -iR 'r57' /var/www/html/
# find /var/www/html/ -name \*.php -type f -print0 | xargs -0 grep c99
# grep -RPn "(passthru|shell_exec|system|base64_decode|fopen|fclose|eval)" /var/www/html/


Your PHP based server is now properly harden and ready to show dynamic webpages. However, vulnerabilities are caused mostly by not following best practice programming rules. You should be consulted further resources for your web applications security needs especially php programming which is beyond the scope of sys admin work.


  1. PHP security - from the official php project.
  2. PHP security guide - from the PHP security consortium project.
  3. Apache suseexec - documentation from the Apache project.
  4. Apache 2.2 - security tips from the Apache project.
  5. The Open Web Application Security Project - Common types of application security attacks.

Recommended readings:

  1. PHP Security Guide: This guide aims to familiarise you with some of the basic concepts of online security and teach you how to write more secure PHP scripts. It's aimed squarely at beginners, but I hope that it still has something to offer more advanced users.
  2. Essential PHP Security (kindle edition): A book about web application security written specifically for PHP developers. It covers 30 of the most common and dangerous exploits as well as simple and effective safeguards that protect your PHP applications.
  3. SQL Injection Attacks and Defense This book covers sql injection and web-related attacks. It explains SQL injection. How to find, confirm, and automate SQL injection discovery. It has tips and tricks for finding SQL injection within the code. You can create exploits using SQL injection and design to avoid the dangers of these attacks.
Please add your favorite php security tool or tip in the comments.
Updated for accuracy!