Wednesday, October 22, 2014

Programming by Voice: Staying Productive without Harming Yourself

One of the reasons I love working at ExtraHop is the lack of meetings and abundance of uninterrupted development time. However, I quickly found after starting that I was unaccustomed to coding for such long periods. A few weeks after I started at ExtraHop, I began to develop discomfort in my wrists and forearms. I have had intermittent trouble with this in the past, but limiting my computer usage at home in the evenings had always been enough to previously solve it. This time, however, was different.
As a very recent college graduate, I was concerned that my daily work activities could be causing permanent injury. I started looking into ergonomic keyboards and mice, hoping to find a cure-all solution. As you might have guessed, I did not find a magical solution, and my situation worsened with each passing week.
While the discomfort was frustrating, I was much more concerned that the injury was preventing me from being able to quickly and easily create and communicate at work and at home.

An Introduction to a Solution

After trying and abandoning several other solutions, a coworker of mine at ExtraHop showed me a PyCon talk by Tavis Rudd, a developer who programs by using his voice. At first, I was skeptical that this solution would be reliable and productive. However, after watching the video, I was convinced that voice input was a compelling option for programmers. Rudd suffered from a similar injury, and he had gone through all of the same investigations that I had, finally determining that a fancy keyboard wasn’t enough to fix it.
That night, I scoured the Internet for people who programmed by voice, looking for tips or tutorials. They were few and far between, and many people claimed that it was impossible. Not easily deterred, I started to piece together a toolkit that would allow me to program by voice on a Linux machine.

Configuration: The Hard Part

It was immediately clear that Dragon NaturallySpeaking was the only option for dictation software. Their product was miles ahead of others in voice recognition, but it only ran on Windows or Mac. Unfortunately I was never successful running Dragon NaturallySpeaking in Wine and had to settle for running in a Windows VM and proxying the commands to the Linux host.
I will leave out some of the configuration steps that I went through in this post. You can find detailed instructions on how to get everything up and running on my GitHub repo.
If you are following along with the instructions, you should now be able to send dictation and the example command to your Linux host, but that will not get you very far with programming. I ended up spending most of the next two weeks writing grammars. The majority of the process was:
  1. Attempt to perform a task (programming, switching windows, etc).
  2. Write a command that would let me do this by voice.
  3. Test that command and add related commands.
  4. Repeat.
The process was slow going, I am hopeful that the repository I linked will help you avoid starting from scratch. Even after using this for about a month, I am still tweaking my commands a couple times a day. Tavis Rudd claims to have over 2000 custom commands, which means that I must still have a long way to go.

The Results

Like Rudd explained in his talk, the microphone is a critical link in this setup. A good microphone that hears only you will make a big difference in both accuracy and speed of recognition. I really like the Yeti from Blue that I am using, but I can generally only use it if the office is mostly quiet.
With the commands I have created so far, I can switch between windows, navigate the web (with the help of Vimium), switch between workspaces, and, most importantly, I can program in Python and Go with decent speed. It is not quite as fast as programming with a keyboard, but it is surprisingly efficient once you learn the commands.
The grammars I have shared in the above GitHub repository are specific to what I need in my workflow. I recommend that you use them as a starting point, while keeping in mind that the computer may recognize words differently for you than it does for me. These grammars are also specific to the languages I use most often. Please don’t hesitate to write ones for your favorite languages. And finally, look for my .vimrc file in my dotfiles repository to find the custom shortcuts that the voice commands trigger.
Coding by voice is not perfect, but it has reached a point where it is a practical option. Don’t continue suffering through wrist and arm discomfort when there is an alternative. Feel free to send me a pull request and we can continue making voice programming better for everyone.

What is a good command-line calculator on Linux

Every modern Linux desktop distribution comes with a default GUI-based calculator app. On the other hand, if your workspace is full of terminal windows, and you would rather crunch some numbers within one of those terminals quickly, you are probably looking for a command-line calculator. In this category, GNU bc (short for "basic calculator") is a hard to beat one. While there are many command-line calculators available on Linux, I think GNU bc is hands-down the most powerful and useful.
Predating the GNU era, bc is actually a historically famous arbitrary precision calculator language, with its first implementation dating back to the old Unix days in 1970s. Initially bc was a better known as a programming language whose syntax is similar to C language. Over time the original bc evolved into POSIX bc, and then finally GNU bc of today.

Features of GNU bc

Today's GNU bc is a result of many enhancements of earlier implementations of bc, and now it comes standard on all major GNU/Linux distros. It supports standard arithmetic operators with arbitrary precision numbers, and multiple numeric base (e.g., binary, decimal hexadecimal) of input and output.
If you are familiar with C language, you will see that the same or similar mathematical operators are used in bc. Some of supported operators include arithmetic (+,-,*,/,%,++,--), comparison (<,>,==,!=,<=,>=), logical (!,&&,||), bitwise (&,|,^,~,<<,>>), compound assignment (+=,-=,*=,/=,%=,&=,|=,^=,&&=,||=,<<=,>>=) operators. bc comes with many useful built-in functions such as square root, sine, cosine, arctangent, natural logarithm, exponential, etc.

How to Use GNU bc

As a command-line calculator, possible use cases of GNU bc are virtually limitless. In this tutorial, I am going to describe a few popular features of bc command. For a complete manual, refer to the official source.
Unless you have a pre-written bc script, you typically run bc in interactive mode, where any typed statement or expression terminated with a newline is interpreted and executed on the spot. Simply type the following to enter an interactive bc session. To quit a session, type 'quit' and press Enter.
$ bc

The examples presented in the rest of the tutorial are supposed to be typed inside a bc session.

Type expressions

To calculate an arithmatic expression, simply type the expression at the blinking cursor, and press Enter. If you want, you can store an intermediate result to a variable, then access the variable in other expressions.

Within a given session, bc maintains a unlimited history of previously typed lines. Simply use UP arrow key to retrieve previously typed lines. If you want to limit the number of lines to keep in the history, assign that number to a special variable named history. By default the variable is set to -1, meaning "unlimited."

Switch input/output base

Often times you want to type input expressions and display results in binary or hexadecimal formats. For that, bc allows you switch the numeric base of input or output numbers. Input and output bases are stored in ibase and obase, respectively. The default value of these special variables is 10, and valid values are 2 through 16 (or the value of BC_BASE_MAX environment variable in case of obase). To switch numeric base, all you have to do is to change the values of ibase and obase. For example, here are examples of summing up two hexadecimal/binary numbers:

Note that I specify obase=16 before ibase=16, not vice versa. That is because if I specified ibase=16 first, the subsequent obase=16 statement would be interpreted as assigning 16 in base 16 to obase (i.e., 22 in decimal), which is not what we want.

Adjust precision

In bc, the precision of numbers is stored in a special variable named scale. This variable represents the number of decimal digits after the decimal point. By default, scale is set to 0, which means that all numbers and results and truncated/stored in integers. To adjust the default precision, all you have to do is to change the value of scale variable.

Use built-in functions

Beyond simple arithmatic operations, GNU bc offers a wide range of advanced mathematical functions built-in, via an external math library. To use those functions, launch bc with "-l" option from the command line.
Some of these built-in functions are illustrated here.
Square root of N:
Sine of X (X is in radians):
Cosine of X (X is in radian):
Arctangent of X (The returned value is in radian):
Natural logarithm of X:
Exponential function of X:

Other goodies as a language

As a full-blow calculator language, GNU bc supports simple statements (e.g., variable assignment, break, return), compound statements (e.g., if, while, for loop), and custom function definitions. I am not going to cover the details of these features, but you can easily learn how to use them from the official manual. Here is a very simple function definition example:
define dummy(x){
return(x * x);

Use GNU bc Non-interactively

So far we have used bc within an interactive session. However, quite popular use cases of bc in fact involve running bc within a shell script non-interactively. In this case, you can send input to bc using echo through a pipe. For example:
$ echo "40*5" | bc
$ echo "scale=4; 10/3" | bc
$ echo "obase=16; ibase=2; 11101101101100010" | bc

To conclude, GNU bc is a powerful and versatile command-line calculator that really lives up to your expectation. Preloaded on all modern Linux distributions, bc can make your number crunching tasks much easy to handle without leaving your terminals. For that, GNU bc should definitely be in your productivity toolset.

Tuesday, October 21, 2014

How to monitor a log file on Linux with logwatch

Linux operating system and many applications create special files commonly referred to as "logs" to record their operational events. These system logs or application-specific log files are an essential tool when it comes to understanding and troubleshooting the behavior of the operating system and third-party applications. However, log files are not precisely what you would call "light" or "easy" reading, and analyzing raw log files by hand is often time-consuming and tedious. For that reason, any utility that can convert raw log files into a more user-friendly log digest is a great boon for sysadmins.
logwatch is an open-source log parser and analyzer written in Perl, which can parse and convert raw log files into a structured format, making a customizable report based on your use cases and requirements. In logwatch, the focus is on producing more easily consumable log summary, not on real-time log processing and monitoring. As such, logwatch is typically invoked as an automated cron task with desired time and frequency, or manually from the command line whenever log processing is needed. Once a log report is generated, logwatch can email the report to you, save it to a file, or display it on the screen.
A logwatch report is fully customizable in terms of verbosity and processing coverage. The log processing engine of logwatch is extensible, in a sense that if you want to enable logwatch for a new application, you can write a log processing script (in Perl) for the application's log file, and plug it under logwatch.
One downside of logwatch is that it does not include in its report detailed timestamp information available in original log files. You will only know that a particular event was logged in a requested range of time, and you will have to access original log files to get exact timing information.

Installing Logwatch

On Debian and derivatives:
# aptitude install logwatch
On Red Hat-based distributions:
# yum install logwatch

Configuring Logwatch

During installation, the main configuration file (logwatch.conf) is placed in /etc/logwatch/conf. Configuration options defined in this file override system-wide settings defined in /usr/share/logwatch/default.conf/logwatch.conf.
If logwatch is launched from the command line without any arguments, the custom options defined in /etc/logwatch/conf/logwatch.conf will be used. However, if any command-line arguments are specified with logwatch command, those arguments in turn override any default/custom settings in /etc/logwatch/conf/logwatch.conf.
In this article, we will customize several default settings of logwatch by editing /etc/logwatch/conf/logwatch.conf file.
Detail =
"Detail" directive controls the verbosity of a logwatch report. It can be a positive integer, or High, Med, Low, which correspond to 10, 5, and 0, respectively.
MailTo =
"MailTo" directive is used if you want to have a logwatch report emailed to you. To send a logwatch report to multiple recipients, you can specify their email addresses separated with a space. To be able to use this directive, however, you will need to configure a local mail transfer agent (MTA) such as sendmail or Postfix on the server where logwatch is running.
Range =
"Range" directive specifies the time duration of a logwatch report. Common values for this directive are Yesterday, Today or All. When "Range = All" is used, "Archive = yes" directive is also needed, so that all archived versions of a given log file (e.g., /var/log/maillog, /var/log/maillog.X, or /var/log/maillog.X.gz) are processed.
Besides such common range values, you can also use more complex range options such as the following.
  • Range = "2 hours ago for that hour"
  • Range = "-5 days"
  • Range = "between -7 days and -3 days"
  • Range = "since September 15, 2014"
  • Range = "first Friday in October"
  • Range = "2014/10/15 12:50:15 for that second"
To be able to use such free-form range examples, you need to install Date::Manip Perl module from CPAN. Refer to this post for CPAN module installation instructions.
Service =
Service =
. . .
"Service" option specifies one or more services to monitor using logwath. All available services are listed in /usr/share/logwatch/scripts/services, which cover essential system services (e.g., pam, secure, iptables, syslogd), as well as popular application services such as sudo, sshd, http, fail2ban, samba. If you want to add a new service to the list, you will have to write a corresponding log processing Perl script, and place it in this directory.
If this option is used to select specific services, you need to comment out the line "Service = All" in /usr/share/logwatch/default.conf/logwatch.conf.

Format =
"Format" directive specifies the format (e.g., text or HTML) of a logwatch report.
Output =
"Output" directive indicates where a logwatch report should be sent. It can be saved to a file (file), emailed (mail), or shown to screen (stdout).

Analyzing Log Files with Logwatch

To understand how to analyze log files using logwatch, consider the following logwatch.conf example:
Detail = High
MailTo =
Range = Today
Service = http
Service = postfix
Service = zz-disk_space
Format = html
Output = mail
Under these settings, logwatch will process log files generated by three services (http, postfix and zz-disk_space) today, produce an HTML report with high verbosity, and email it to you.
If you do not want to customize /etc/logwatch/conf/logwatch.conf, you can leave the default configuration file unchanged, and instead run logwatch from the command line as follows. It will achieve the same outcome.
# logwatch --detail 10 --mailto --range today --service http --service postfix --service zz-disk_space --format html --output mail
The emailed report looks like the following.

The email header includes links to navigate the report sections, one per each selected service, and also "Back to top" links.
You will want to use the email report option when the list of recipients is small. Otherwise, you can have logwatch save a generated HTML report within a network share that can be accessed by all the individuals who need to see the report. To do so, make the following modifications in our previous example:
Detail = High
Range = Today
Service = http
Service = postfix
Service = zz-disk_space
Format = html
Output = file
Filename = /var/www/html/logs/dev1.html
Equivalently, run logwatch from the command line as follows.
# logwatch --detail 10 --range today --service http --service postfix --service zz-disk_space --format html --output file --filename /var/www/html/logs/dev1.html
Finally, let's configure logwatch to be executed by cron on your desired schedules. The following example will run a logwatch cron job every business day at 12:15 pm:
# crontab -e
15 12 * * 1,2,3,4,5 /sbin/logwatch
Hope this helps. Feel free to comment to share your own tips and ideas with the community!