Monday, July 27, 2015

How to enable logging in Open vSwitch for debugging and troubleshooting

Question: I am trying to troubleshoot my Open vSwitch deployment. For that I would like to inspect its debug messages generated by its built-in logging mechanism. How can I enable logging in Open vSwitch, and change its logging level (e.g., to INFO/DEBUG level) to check more detailed debug information? Open vSwitch (OVS) is the most popular open-source implementation of virtual switch on the Linux platform. As the today's data centers increasingly rely on the software-defined network (SDN) architecture, OVS is fastly adopted as the de-facto standard network element in data center's SDN deployments.
Open vSwitch has a built-in logging mechanism called VLOG. The VLOG facility allows one to enable and customize logging within various components of the switch. The logging information generated by VLOG can be sent to a combination of console, syslog and a separate log file for inspection. You can configure OVS logging dynamically at run-time with a command-line tool called ovs-appctl.

Here is how to enable logging and customize logging levels in Open vSwitch with ovs-appctl.
The syntax of ovs-appctl to customize VLOG is as follows.
$ sudo ovs-appctl vlog/set module[:facility[:level]]
  • Module: name of any valid component in OVS (e.g., netdev, ofproto, dpif, vswitchd, and many others)
  • Facility: destination of logging information (must be: console, syslog or file)
  • Level: verbosity of logging (must be: emer, err, warn, info, or dbg)
In OVS source code, module name is defined in each source file in the form of:
For example, in lib/netdev.c, you will see:
which indicates that lib/netdev.c is part of netdev module. Any logging messages generated in lib/netdev.c will belong to netdev module.
Depending on severity, several different kinds of logging messages are used in OVS source code: VLOG_INFO() for informational, VLOG_WARN() for warning, VLOG_ERR() for error, VLOG_DBG() for debugging, VLOG_EMERG for emergency. Logging level and facility determine which logging messages are sent to where.
To see a full list of available modules, facilities, and their respective logging levels, run the following commands. This command must be invoked after you have started OVS.
$ sudo ovs-appctl vlog/list

The output shows the debug levels of each module for three different facilities (console, syslog, file). By default, all modules have their logging level set to INFO.
Given any one OVS module, you can selectively change the debug level of any particular facility. For example, if you want to see more detailed debug messages of dpif module at the console screen, run the following command.
$ sudo ovs-appctl vlog/set dpif:console:dbg
You will see that dpif module's console facility has changed its logging level to DBG. The logging level of two other facilities, syslog and file, remains unchanged.

If you want to change the logging level for all modules, you can specify "ANY" as the module name. For example, the following command will change the console logging level of every module to DBG.
$ sudo ovs-appctl vlog/set ANY:console:dbg

Also, if you want to change the logging level of all three facilities at once, you can specify "ANY" as the facility name. For example, the following command will change the logging level of all facilities for every module to DBG.
$ sudo ovs-appctl vlog/set ANY:ANY:dbg

How to run DOS applications in Linux

Chances are that most of you reading along those lines have started your “adventure” in computers through DOS. Although this long deprecated operating system is only running in our memories anymore, it will always hold a special place in our hearts. That said, some of you may still want to drink a sip of nostalgia or show your kids what old days were like by running some MS-DOS applications on your Linux distribution. The good news is, you can do it without much effort!
For this tutorial, I will be using a DOS game I was playing when I was a little kid called “UFO Enemy Unknown”. This was the first ever squad turn-based strategy game released by Microprose a bit over twenty years ago. A remake of the game was realized by Firaxis in 2012, clearly highlighting the success of the original title.


Since DOS executables are .exe files, it would be natural to think that you could run them with wine, but unfortunately you can't. The reason is stated as “DOS memory range unavailability”.
What this means is that the Linux kernel forbids any programs (including wine) from executing 16-bit applications and thus accessing the first 64k of kernel memory. It's a security feature and it won't change, so the terminal prompt to use DOSBox can be the first alternative option.


Install DOSBox from your Software Center and then open your file manager and make sure that you create a folder named “dosprogs” located in your home directory. Copy the game files inside this folder and then open dosbox by typing “dosbox” in a terminal. Now what we need to do is to mount the “dosprogs” folder into dosbox. To do this type mount c ~/dosprogs and press enter on the DOSBox console. Then type c: to enter the newly mounted disk as shown in the following screenshot.
You may then navigate the disk folders by using the “cd” command combined with the “dir” until you locate the game executable. For example, type “cd GAME” to enter the GAME folder and then type “dir” and press enter to see what the folder GAME contains. If the file list is too long to see in a screen, you may also give the “dir /w/p” command a try. In my case, the executable is UFO.bat and so I can run it by typing its name (with the extension) and pressing enter.


Another application that allows you to run DOS executables under Linux is DOS Emulator (also available in the Software Center). It is more straight forward in regards to the mounted partitions as you simply type “D:” and enter on the console interface to access your home directory. From there you can navigate to the folder that contains the DOS executable and run it in the same way we did in DOSBox. The thing is though that while DOSemu is simpler to use, it may not run flawlessly as I found through my testing. You can always give it a try though and see how it goes.

Bash: Find out the exit codes of all piped commands

How do I get exit status of process that's piped to another (for e.g. 'netstat -tulpn | grep nginx') on a Linux or Unix-like system using a bash shell?

A shell pipe is a way to connect the output of one program to the input of another program without any temporary file. The syntax is:
Tutorial details
DifficultyEasy (rss)
Root privilegesNo
Estimated completion time2m
command1 | command2 | commandN
command1 | filter_data_command > output
get_data_command | verify_data_command | process_data_command | format_data_command >

How to use pipes to connect programs

Use the vertical bar (|) between two commands. In this example, send netstat command output to grep command i.e. find out if nginx process exits or not in the system:
# netstat -tulpn | grep nginx
Sample outputs:
Fig.01: Find the exit status of pipe command
Fig.01: Find the exit status of pipe command

How to get exit status of process that's piped to another

The syntax is:
command1 | command2
echo "${PIPESTATUS[@]}"
command1 | command2
echo "${PIPESTATUS[0]} ${PIPESTATUS[1]}"
PIPESTATUS is an array variable containing a list of exit status values from the processes in the most-recently-executed foreground pipeline. Try the following commands:
netstat -tulpn | grep nginx
echo "${PIPESTATUS[@]}"
true | true
echo "The exit status of first command ${PIPESTATUS[0]}, and the second command ${PIPESTATUS[1]}"
true | false
echo "The exit status of first command ${PIPESTATUS[0]}, and the second command ${PIPESTATUS[1]}"
false | false | true
echo "The exit status of first command ${PIPESTATUS[0]}, second command ${PIPESTATUS[1]}, and third command ${PIPESTATUS[2]}"
Sample outputs:
Fig.02: Use the PIPESTATUS array variable to get the exit status of each element of the pipeline
Fig.02: Use the PIPESTATUS array to get the exit status of each element of the pipeline (click to enlarge)

Putting it all together

Here is a sample script that use ${PIPESTATUS[0]} to find out the exit status of mysqldump command in order to notify user on screen about database backup status:
### Purpose: : Backup database ###
### Author: Vivek Gite , under GPL v2.x+ or above. ###
### Change as per your needs ###
# Failsafe? Create dir #
[  ! -d "$DEST" ] && $MKDIR -p "$DEST"
# Filter db names
DBS="$($MYSQL -u $MUSER -h $MHOST -p$MPASS -Bse 'show databases')"
DBS="$($SED -e 's/performance_schema//' -e 's/information_schema//' <<<$DBS)"
 # Okay, let us go
 for db in $DBS
                 tTime=$(date +"${NOWFORMAT}")
                 $MYSQLDUMP -u $MUSER -h $MHOST -p$MPASS $db | $GZIP -9 > $FILE
    if [ ${PIPESTATUS[0]} -ne "0" ];
        echo "The command $MYSQLDUMP failed with error code ${PIPESTATUS[0]}."
        exit 1
        echo "Database $db dump successfully."

A note about zsh user

Use the array called pipestatus as follows:
true | true
echo "${pipestatus[1]} ${pipestatus[2]}"
0 0