There will be a meeting Saturday, May 25th, 2013, 10:00am, at the Pleasant Ridge Branch of the Cincinnati Public Library, located at 6233 Montgomery Road, Cincinnati, OH

Google Maps Goodness

I don’t have a topic yet, if there’s something you want to hear about, let me know, but leave out the Spam!
Steve Jones E-Mail

Thank you and hope to see you at the meeting!

Steve Jones

Automating Simple Tasks on Linux (Shell Scripts and other Simple Tools) Shells

Monty Stein Mar 24, 2000

There are 2 major groupings of shells in common use. Bourne derived shells evolved from the first Unix shell. Bourne (sh), Korn (ksh) and Bourne Again Shell (bash) are the major variants in use. The other evolutionary branch are the C shells from the early Berkeley Unix systems. csh is the form that most people see. Apparently it is a necessary feature of any shell to have the name form a pun.

Everything below will use Bourne style syntax since it will work on the broadest set of shell programs.

When using a shell it isn’t directly apparent how much power you have at your fingertips. There is a language that shells understand (embodied in shell scripts) that is also available when you are typing in front of one in interactive mode or put into files as scripts.
Globbing is the step that is performed on wildcards in filenames to expand them into real filenames. Thus *.log is expanded to a list of all the files in the directory that match the pattern.

matches any number of characters
matches any one characters
matches a range of characters (like [0-9] for all the numbers)

For example:

echo * # when you don’t have enough of
# a system left to run ls

There are 3 I/O streams that are set up by the shell for any program that it starts. Standard Input (stdin or stream number 0), Standard Output (stdout or #1), and Standard Error (stderr or #2). Normally the program sees it’s input from stdin, sends output to stdout and reports any errors to stderr. These streams can be manipulated before the program sees them by redirecting them. Normal forms of redirection are simply to and from files:

someprog output_file

In this case the input and output are handled by files and any errors would likely be reported to the screen. (Programs, of course, are not limited to use only these streams. However, most shell programming uses support programs of this form.)
Streams can be joined by referencing them by number. Thus:

someprog output.log 2>&1

joins stderr (#2) and stdout (#1) together and put them into output.log. Ordering is critical with this form. The steps read from right to left.

There is one more form of stream and it is called a “Here Document”. Its use is rare. Check the manual page for more information.
Environment variables are most commonly used to pass some tidbit of information to a program that needs login or machine specific configuration (such as what X display your programs should display to).

To set a variable, the name of the variable is used with an equals sign immediately following (naming convention for environment variables is to use upper case). To access the contents of the variable, use a dollar sign and then the variable name:

/tmp-> VAR=contents
/tmp-> echo $VAR

In places where the name of the variable would touch another alphanumeric character, the variable can be bracketed by curly braces to force the correct behavior:

mv $VAR ${VAR}old

(in this case, the shell would be looking for a variable VARold if the braces were not used)

Variables can be exportable or not. Exported variables are passed to programs that the shell starts (and to any that they start). Unexported variables are restricted to the current shell. Any number of variables can be exported with the export command:

/tmp-> VAR=contents
/tmp-> sh -c 'echo $VAR' # start a subshell as another process

/tmp-> export VAR
/tmp-> sh -c 'echo $VAR' # start a subshell with the exported value

Remember, under the design of the Unix process model, programs spawned by the shell cannot set variables in the parent shell. The way around this is to source a file in the current shell:

/tmp-> . somefile

would run somefile as if it was typed in at that point.


will print all the variables that the shell knows about.
without any arguments will print all exported variables.
will remove a variable.

Special Variables

return code of the last program
PID of last background process spawned
PID of the current shell, useful for creating temp files
all the arguments to this shell
what this shell was called by
$1 … $9
Arguments to the shell script (more args are there, just call shift to get to them)

The shell supports a rich variety of common quoting types (all right, 5).
Double quotes bracket strings and allow variables to be expanded.
Single quotes bracket strings and do not allow any shell string operations inside.
Back quotes will run the contents in a separate shell and return the output IN PLACE.
A backslash will quote a single character.
A pound sign “#” will comment out the rest of the line

/tmp-> VAR=contents
/tmp-> echo "this string has $VAR"
this string has contents
/tmp-> echo "this string has \$VAR"
this string has $VAR
/tmp-> echo 'this string has $VAR'
this string has $VAR
/tmp-> echo `echo $VAR|tr a-z A-Z`

Job Control
Programs can be run in the background by putting an ampersand “&” at the end of the line. Last backgrounded job process ID is in the “!” variable. wait will wait on all background jobs to finish when called with no arguments, or by PID when it is given as the argument. Wait returns the exit code of the process that it waited on.
Testing and Control Structures
Commands that have completed successfully return a value of 0. This way errors can be identified by a rich set of return codes (thus TRUE is equal to 0 and FALSE is everything else). The last command’s return value is stored in the “?” variable.

The test command sets its return code based on an expression that can return information about a file or compare strings. The test command itself can be accessed by the “[” “]” pair. (For the historically minded: The original Bourne shell didn’t have the square bracket shortcuts and the test program had to be called directly. That is why if you want to get a list of all that test can test you have to run man test.)
A call to test would look something like:

[ -r /tmp/output ]

This would read: If the file /tmp/output is readable, return a TRUE exit status.
Note: When testing the contents of variables, always put them in double quotes. This avoids the problem of:

[ $VAR = junk ]

The shell sees:

[ = junk ]

if $VAR is quoted, the empty string would be visible to the shell.
Check the manual page for test for all the different options that it takes. Note: Some of the test options are platform dependent. Keep them simple for portability sake, use -r (readable) instead of -e (exists) which does not work with HP-UX’s test.

if expression
elif expression

The classical if statement. If the result of expression is TRUE, then execute the then block. The expression can be a call to test or any other program.

if [ ! -d tempdir ] # create a temporary directory
then # if it doesn't already exist
mkdir tempdir

Lazy Evaluation
Lazy evaluation takes advantage of the fact that if A and B must be true in order for something to happen and A is not true, there isn’t any point in evaluating B (the converse for “or” also applies, it just flips the logic). It can shorten simple tests in scripts.
For example: This:

if [ -r somefile ]
cat somefile

Will run identically to:

[ -r somefile ] && cat somefile

For “or” the logic inverts. If the first command is not TRUE, execute the second:

[ $RETURNCODE -eq 0 ] || echo "command failed"

A case statement allow simplification of lots of nested if/then blocks. It takes the form of:

case value in

The patterns are matched against the value using the same expansion rules that would be used for filename globbing.
Functions encompass small, often called portions of a script. From the outside the function looks like another program. From the inside the function everything looks like it is running in a separate shell except that all the variables from the parent script are available to the function for reading and writing.
The return (exit) value is passed with a call to return.

Functions look like:
function somefunc {
echo "hello $1"
return 0

And are called like:

somefunc "world"

For loops iterate over a list of values and execute a block for each entry in the list. They look like

for variable in list

list can be the output from a program or a globbed list. For example:

for filename in *.c
cp $filename $filename.backup

While loops look and act a lot like for loops, but will loop indefinitely until an expression returns FALSE. It looks like:

while expression

For example:

while [ ! -r "STABLE" ]
echo "waiting on STABLE flag"
sleep 60

Common Commands
ls will LiSt the contents of a directory. Since most people learn about this command fairly quickly, I’ll focus on the more useful flags:
-S sort by size (GNU only)
-t sort by time last modified
-r reverse sort
-a all files
-d do not enter directory
cut will cut lines of text by column or by delimiter.
-c10-24 would output columns 10 through 24
-d: -f1,3 would output the second and fourth fields delimited by a colon

this:that:the other


this:the other

Stream EDitor. Will run ed style commands on files. The most common way to use it is for search and replace on the fly.

the first line

sed s/first/second/g returns

the second line

character TRanslator. tr can translate one set of characters into another as well as suppress duplicate input characters.

lower case

tr a-z A-Z returns


the -s switch will force the suppression of duplicated sequences of characters

this that the other

tr -s ‘ ‘ returns

this that the other

(useful to preprocess a tabular report into something that cut can work on) The -d switch will delete characters (can be very useful with the -c complement switch to return only a given set of characters).

wc -l somefile|tr -cd “0-9” # gives the number of lines
# w/ no other chars

sorts files. The most common way to use this is sort -u to suppress duplicated lines after the sort. A -r will reverse the sort, -n will attempt to convert string based numbers into machine numbers for the sort.
Find recursively descends into filesystems and (in the simplest form) prints filenames based on certain criteria.

find . -type f -print

will print the names of all files below the current directory

find . -newer /etc/lastbackup -type d -print

will print the names of all directories that have had files added or deleted since the file /etc/lastbackup was last modified.
xargs will build up command lines from standard input. When supplied a command to run, it will execute that command with as many arguments built up from it’s input as the OS will allow. -n num will limit the number of args passed to each command invocation

find . -type f -print|xargs grep “your keys”

would search all files below and in the current directory for the string “your keys”
Running Jobs at Certain Times
The crond daemon runs once a minute and runs any jobs scheduled by crontab or at. It normally handles all the recurring jobs that maintain the system. It can also be a huge security hole (there was a notable problem with the vixiecron system in RH5 series). Because of the problems that it use can cause, the cron system has built in to it a way of restricting its use by the allow and deny files that are stored in /etc.
Each user (that is allowed to) has a crontab file that is read/written with the crontab command. Crontabs are used for jobs that need to be run at regular recurring points in time. The crontab file has this structure:

minute hour month-day month weekday job

So, to fetch mail using a script called /usr/local/bin/getmymail every minute during business hours:

* 7-17 * * 1-5 /usr/local/bin/getmymail

Read as: for every minute between 7am to 5pm from Monday (day 1) to Friday (day 5) run the job /usr/local/bin/getmymail.

Use crontab -l to get the contents of your crontab entries. It is a very good idea to keep a master copy that you can edit and reload.

A possible edit session would be:

crontab -l >mycrontabfile

edit mycrontabfile

crontab mycrontabfile

The scripts that are run will not have any variables but the minimal user environment set. Any scripts that are run should set up any variables that they need (an expanded $PATH variable for example) or assume nothing about the environment they will be running in. Any output generated will be mailed to the user.
Runs a job at a specific time. Differs from crontab in that it will run the job only once and that all environment variables are carried through from the shell that called it. The script is to come in via standard input any output will be mailed to the user.

At allows easy setting of the time that the script is to be run:

echo “myscript”|at now + 5 hours

would run myscript 5 hours from now.

echo “someotherscript”|at 5 pm Tuesday

would run someotherscript at the next 5pm on a Tuesday. Be certain to double check the date that at reports when the job is scheduled so that it is what you expected.

At will also run jobs by date:

echo “were you fooled?”|at 5 pm april 1

at -l will list all pending jobs.
atrm (or at -r on some systems) will remove a numbered job.
Close to at now, but holds the job until the load average falls below 0.8 as well as running the job at low priority. Play nice.


September 27, 1998
The incorporated organization shall be named the Cincinnati GNU/Linux Users Group, Incorporated, also known as the Cincinnati Linux Users Group, hereinafter known as CLUG.
The object of CLUG is to bring together computer users who encourage the study of and use of the GNU/Linux operating system. Education of the members and the general public is a goal of the organization.

Section 1. Membership in the organization shall be open to all who are interested in the GNU/Linux operating system.

Section 2. Classes of membership

Individual, shall pay full membership dues.
Family membership shall include all rights of membership with the understanding that a family membership shall be entitled to one vote, although those having family memberships shall be permitted to make motions and speak in debate, making a dues payment as established for an individual membership.
Student membership, which may result in a reduced dues rate.

Section 3. Dues shall be proposed by the Board of Directors to be approved by the general membership.

The Board of Directors may reduce the dues for a membership on request, determining such reduction in executive session.
Family memberships shall be available to those residing in the same household.

Section 4. The fiscal year begins annually on January 1.

Dues are to be paid on January 1. Dues shall be delinquent if not received by the treasurer by February 1. Members whose dues are delinquent shall be notified of delinquency within two weeks and members whose dues are delinquent after February 28 shall be dropped from membership.
Membership dues shall be prorated on a monthly basis.


Section 1. The Board of Directors shall serve as trustees of the corporation.

Section 2. The officers shall be a president, vice- president, secretary, and treasurer.

Section 3. There shall be three directors who shall serve with the officers on the Board of Directors.

Section 4. Officers and directors shall be voting members of CLUG; they shall serve one-year terms or until their successors are elected and begin their terms of office. No officer shall serve more than two successive terms in the same position.

Section 5. Election shall take place in November with term of office to begin January 1.

Section 6. A nominating committee of three persons shall be selected by the Board with the chairman of the nominating committee not being a member of the Board. They shall nominate at least one person for each office and directorship and communicate the nominations by mail or electronic means to the membership by October 15. Nominations from the floor shall be accepted.

Section 7. Elections shall be held at the November meeting, with election by ballot unless there is only one nominee for a position when a voice vote may elect.

Section 8. The Board shall meet on call of the president or any two members with notice of at least seven days.

Section 9. The Board shall have general supervision of CLUG between meetings of the general membership and shall be authorized to determine issues which may be deliberated and voted on by electronic means. Quorum of the Board shall be four members.

Section 10. Vacancies in any position of the Board of Directors shall be filled by those remaining on the Board of Directors with the exception of the office of president which shall be filled by the vice-president should that office become vacant.

Section 11. Board of Directors meetings are open to members. The Board may go into executive session for deliberation of sensitive issues.

Section 1. Regular meetings shall be held monthly with the location, time, and date to be determined by the Board of Directors who have the authority to postpone or cancel meetings upon the agreement of four members of the Board with notice by electronic means.

Section 2. The regular meeting in November shall be the annual meeting at which officer and committee reports shall be heard, and election held. Meeting notice of at least fourteen days shall be given to all members by mail or electronic means.

Section 3. Special meetings may be held on the call of four members of the Board, or ten members who request such a meeting with notice of at least seven days by mail or electronic means.

Section 4. The quorum for regular meetings of the membership is ten members.

Section 5. Regular meetings are open to the public.

Section 1. Duties of the officers are as follows:

President: The President shall preside at all meetings of the membership and the Board of Directors. The President shall call the regular meetings and special meetings of the organization. The President shall enforce all rules of the organization and perform all duties of the presidents of like organizations. The President is a member ex-officio of all committees except the nominating committee.
Vice-president: The Vice-president shall perform all duties of the President in case of absence or disability of the President.
Secretary: The Secretary shall record minutes of all meetings of the general membership and the meetings of the Board of Directors.
Treasurer: The Treasurer shall assume responsibility for all funds and financial records of the organization. The Treasurer shall issue financial reports at meetings of the general membership and of the Board of Directors.

Section 2. In addition to described duties, officers shall perform other duties customary to the office.
The Board of Directors or the general membership gathered in meetings are each empowered to establish committees and special interest groups as needed.
The rules contained in the current edition of Robert’s Rules of Order Newly Revised shall govern CLUG in all cases to which they are applicable and in which they are not inconsistent with these bylaws and any special rules of order CLUG shall adopt.
These bylaws may be amended at any meeting by a two-thirds vote provided notice of proposed changes, additions, or deletions shall have been submitted in writing thirty days before the meeting. Notice shall be mailed or be sent by email or FAX to all members, at least two weeks before the meeting.

Backing up VirtualBox VMs

This script below is what I use to make backups of running Windows VMs, they are put into hibernation, then the drive is imaged, then the machine is woken back up. There is downtime with this method, but not much and it is easy to schedule as a low use-time service.

## Destination of backup files
## Script won't run if directory doesn't exist, and please note that this runs with a normal
## users privileges, so the $BACKUPDEST must be writable by the user running the VM.

## How many days a compressed version will be left on server

## Exempt VMs - Place each VM with a space seperating them
## Example: ( linux winxp win7 ) or
## ( "Ubuntu 8.04" "Windows XP" ) or
## ( None )
## More here:
## Run Command to find list of VMS:
## VBoxManage list vms | grep '"' | cut -d'"' -f2 2>/dev/null

## No need to modify below here
DATEFILE=`/bin/date +%Y%m%d`
VMLIST=`VBoxManage list vms | grep '"' | cut -d'"' -f2 2>/dev/null`

## Functions

## Notify the starting time of backup
function startScript {
echo "-----------------------------------------------------"
echo "START - ${VM}"
echo "Host: ${HOST}"
echo "Date: `date`"
echo "-----------------------------------------------------"

## Create the backup directories if they do not exist
function doCheckDirectories {
## Check to see if BACKUPDEST exist
if [ ! -d ${BACKUPDEST} ]; then
echo "BACKUPDEST does not exist!! Exiting Program."
exit 0
## If the archives directory does not exist, create it
if [ ! -d ${BACKUPDEST}/archives ]; then
echo "${BACKUPDEST}/archives directory does not exist, creating . . ."
mkdir "${BACKUPDEST}/archives"
## If the directories directory does not exist, create it
if [ ! -d ${BACKUPDEST}/directories ]; then
echo "${BACKUPDEST}/directories directory does not exist, creating . . ."
mkdir "${BACKUPDEST}/directories"

## If this VM is in our exempt array, set VM_EXEMPT to skip entirely.
function doCheckExempt {
## array, if we get a match, set VM_EXEMPT to true
for check_vm in "${EXEMPTION_ARRAY[@]}"; do
if [ "${check_vm}" = "${VM}" ]; then
echo "${VM} is on the exception list, skipping."

## Suspend VM if its running, skip it if not
function suspendVM {
## Check state of VM
VMSTATE=`VBoxManage showvminfo "${VM}" --machinereadable | grep "^\(VMState=\)" | cut -d'"' -f2 2>/dev/null`

echo "${VM} state is currently: ${VMSTATE}"

## If VM is running, suspend it, otherwise, move on
if [ "${VMSTATE}" = "running" ]; then
echo "Suspending ${VM} . . ."
## Stop vm by saving current state (pause|resume|reset|poweroff|savestate)
VBoxManage controlvm ${VM} savestate 2>/dev/null
[ $? ] && echo Success || echo Failure
echo "${VM} Suspended on `date`"
echo "${VM} was not running, not suspending - `date`"

## Backup VM
function doBackup {
## Display location of XML file
XMLFILE=`VBoxManage showvminfo "${VM}" --machinereadable \
| grep "^\(CfgFile=\)" \
| cut -d'"' -f2 2>/dev/null`
## Display location of vdi file
## LOCATION=`VBoxManage showvminfo "${VM}" --machinereadable | grep ".vdi" | cut -d'"' -f4 2>/dev/null`
## Display location of vdi or vmdk file
LOCATION=`VBoxManage showvminfo "${VM}" --machinereadable \
| grep -e ".vdi" -e ".vmdk" \
| cut -d'"' -f4 2>/dev/null`

## If the directories directory does not exist, create it
if [ ! -d ${BACKUPDEST}/directories/${VM} ]; then
echo "${BACKUPDEST}/directories/${VM} does not exist, creating . . ."
mkdir "${BACKUPDEST}/directories/${VM}"
## Backup VM (clonehd)
echo "Backing up "${VM}" to ${BACKUPDEST}/directories/${VM}/"
rsync --inplace -a --stats "${XMLFILE}" "${BACKUPDEST}/directories/${VM}/"
[ $? ] && echo Success || echo Failure
rsync --inplace -a --stats "${LOCATION}" "${BACKUPDEST}/directories/${VM}/"
[ $? ] && echo Success || echo Failure

## Start VM if suspened
function doStart {
if [ "${VMSTATE}" = "running" ]; then
echo "Starting ${VM} . . ."
## Resume VMs which were running [--type gui|sdl|vrdp|headless]
VBoxManage startvm ${VM} --type headless
echo "${VM} Resumed on `date`"
[ $? ] && echo Success || echo Failure
echo "${VM} was not running, not resuming - `date`"

## Making a compressed and mobile backup
function doTar {
echo "taring up ${VM} to ${BACKUPDEST}/archives/${fileName}"
tar -czf "${BACKUPDEST}/archives/${fileName}" "${BACKUPDEST}/directories/${VM}" 2>/dev/null
[ $? ] && echo Success || echo Failure

## Clean up any tars or logs that are older than DAYS_TO_KEEP_TAR
function doCleanTar {
echo "Cleaning up tars older than ${DAYS_TO_KEEP_TAR} day(s) old"
find "${BACKUPDEST}/archives" -name "backup_${VM}*.tgz" -mtime ${DAYS_TO_KEEP_TAR} -exec rm -vf {} \;
[ $? ] && echo Success || echo Failure
echo "Cleaning up logs older than ${DAYS_TO_KEEP_TAR} day(s) old"
find "${BACKUPDEST}/" -name "*-log" -mtime ${DAYS_TO_KEEP_TAR} -exec rm -vf {} \;
[ $? ] && echo Success || echo Failure

## Notify the finishing time of backup
function finishScript {
echo "-----------------------------------------------------"
echo "FINISH - ${VM}"
echo "Host: ${HOST}"
echo "Date: `date`"
echo "-----------------------------------------------------"

## Script

## Make sure we have the appropriate directories for backups
## Start loop
for VM in ${VMLIST}; do
sleep 1
## Check exempt list
if [ "$VM_EXEMPT" = "false" ]; then
## Suspend VM
sleep 3
## Do Backup
## Start if suspended
## Compressing backup
sleep 3
## Clean old backups
sleep 3
## Reset exemption
done >> ${BACKUPDEST}/${DATEFILE}-log

Rsync to Fat32 drives

I regularly provide one of my clients with a backup of his data in the form of an external hard drive. Since his server runs CentOS Linux and his computer is a Windows machine, I need to provide a drive formatted Fat32 so he can plug it into his computer and access the data without problem.

# mount -t vfat /dev/sdc1 /mnt/usb -o shortname=mixed -o utf8

The “shortname=mixed” keeps the case preserved, as otherwise vfat will convert any filename that’s 8 characters or less to lower case (default behavior is “shortname=lowercase”) and cause problems for rsync. UTF8 is what Windows uses when mounting filesystems, so we specify that to ensure that we’re mounting it the same way (default is to mount iso-8859-1, even though the underlying vfat filesystem will store filenames in UTF8 format).

My normal mirror command, “rsync -az /home /mnt/usb“, doesn’t work because -a is a shortcut for the following options:

-a, –archive archive mode; same as -rlptgoD (no -H)
-r, –recursive recurse into directories
-l, –links copy symlinks as symlinks
-p, –perms preserve permissions
-t, –times preserve times
-o, –owner preserve owner (super-user only)
-D same as –devices –specials
-g, –group preserve group

Using -o will cause errors, as rsync will copy the file and then chown (change owner) the file. Fat32 doesn’t support Unix owership or permissions, so rsycn will error on every file that is copied. Ditto for -p and -g. Symlinks aren’t supported either, and we don’t want -L to copy the destination file of the symlink (that will produce multiple copies of a file/directory, not desirable in this particular instance). The -D option is irrelevant because we are only copying website data, so we don’t need special devices (/dev/*).

That leaves -r (recursive), -t (preserve times) for our vfat options. There’s no need to use compression (-z) since we’re not syncing across the network.

So the best command to copy form ext3 to Fat32 drive is something like this:

rsync -rtv /home /mnt/usb

I like using -v for verbosity, unless I’m running this within a shell script.

A good reference for further reading on Fat32 with Linux: