Some shell script tricks

It’s easy to forget to check for errors, consider

What if $1 is not a directory?

In its current form, the cd command might fail, and the line after that statement will get executed. This can have nasty side effects, like deleting or creating files inside the wrong directory.

Instead of checking every return code of every command, which would increase the complexity and reduce the readability of the script, it is possible to stop and exit immediately on the first failure:

This option is available on POSIX sh, which means it is available on very shell that claims to be compatible: bash, dash, zsh, …​

Of course, there are situations where we can and want to handle errors, the documentation states it clearly:

Knowing that on every error the script terminates

If you are using bash instead of POSIX sh, you might also want to enable the following option: set -o pipefail Unfortunately, it is not available on all shells, and there is no portable solution.

shellcheck is a must-have when writing some scripts.

It does not support zsh, but you probably should prefer writing scripts in bash or in sh, given that those are available on nearly all systems, and given that zsh does not really offer great benefits over bash for non-interactive shells (ie: scripts).

It is a static analyzer, thus it does not need to execute the script to find common pitfalls.

Consider again

What if it is a directory but it contains spaces? Then the cd command will be executed with multiple parameters and probably fail.

What if the directory name contains *? The shell might expand it to other characters, thus cd might change to the wrong directory.

What if the directory is named -? While it is difficult to foresee the third problem, the first two have easy fixes but let’s not digress.

For example, in the previous example, it would have mentioned that we should write cd "$1" instead of cd $1 to avoid issues if $1 holds a valid path with spaces or to avoid globbing.

When dealing with optional parameters, dealing with unset and set but empty variables is often confusing.

Adding

Will help to spot some edge cases, since it will trigger an error, and together with set -o errexit the script will stop immediately.

In my experience, a not-so-well-known (or used) feature of the shell is parameter expansion.

Using shell expansion is the easiest way to provide default values:

For something more complex, test if the value is set

Sure, it’s nice to write ls -a instead of ls --all, but for more exotic commands (do you remember all tar options?), writing down the long-form improves readability. And since debugging shell scripts is not that nice, those should be as easy as possible to read.

I normally do not know all the options for a given command. Searching on the man page, or online with a search engine, for -a will normally give less meaningful results than searching for --all, especially for more esoteric commands.

For commands written directly in the console and executed immediately, it does not make much sense to prefer the long form to the short one if both are known at the given time. But scripts might be read by other people or proper self in a week, so they should be much more self-explanatory.

As in the previous example, if you have a directory named -, you cannot simply cd to it, since cd - will bring you to your previous directory. If you do not have an absolute path, prepending ./ to any unknown relative path, or converting any unknown relative path to an absolute will fix the issue.

cd is not the only command or tool that has such problems, many command line programs accept arguments and directories as arguments, and if a directory name coincides with another argument, it gets tricky to distinguish between those.

For example mkdir -la and mkdir ./-la creates the same directory, but ls -la and ls ./-la will do different things.

Therefore I strive to create "portable" names when creating files or directories. The POSIX specification defines a "Portable Filename Character Set" (not to be confused with the "Portable Character Set"). Long story short: only ASCII letters, numbers, period, underscore, and minus (hyphen) are accepted for a portable filename.

Notice that with this definition it is still possible to create a filename that begins with a minus, and could thus get confused with a command line option. It is therefore better to create portable filenames that do not begin with a hyphen for simplicity.

On the other hand, spaces are not part of the "Portable Filename Character Set". Since it is such a common character all tools should be able to handle a file or path with a space in it.

Nevertheless, we still have to deal with existing files, and those might have strange names. Prepending ./ (or converting the relative path to an absolute) fixes the problem in most cases since most of the time only directories beginning with - is a problem. I’ve never seen, for example, a program using something beginning with ./ as an argument and thus with possible issues between filenames and parameters.

Notice that also tools like realpath (for converting relative paths to absolute) can take arguments. On my machine, the following commands fail; realpath -e. On the other hand realpath ./-e gets converted correctly.

Some tools give the possibility to split arguments and paths with --. If such an option is available, use it, especially in a script, since it reduces a lot of hidden bugs. For example realpath -- -e works correctly.

Otherwise, try not to rely on anything. Even the encoding of filenames is generally unspecified. This is one of the many reasons why the output of ls should not be parsed.

Some scripts are merely one-liners, whereas others are so complex that it makes sense to split them into functions.

Personally, I like to have a main function (like in many other languages), this makes it easy to find the entry point, and eventually split the script into more modules. When executing the script, this line forwards all arguments unchanged to the main function: main "$@"

Since the script is complex (otherwise no need to add a main function), main should call other functions that contain the business logic.

Another common function is help. It prints a message that explains what the script does, and what are possible options. This function is normally executed when the script is called with -h, --help or used incorrectly.

It is an accepted practice to change environment variables, but most of the time there is no need.

Changing the global status might have unpredictable side effects, therefore it is better to try to limit the scope as much as possible.

Instead of writing

which might be problematic if a was already defined, we can write

which will nearly do the same thing, and without changing the global scope.

The same holds for common global variables:

Otherwise, it is possible to try to backup and restore the global status

It surely works in simple cases, but with more complex logic resetting the old state is error-prone. For example command_or_function might change IFS too and options like set -u could cause a failure when backing IFS up if it is not set.

Another global state is the current directory. Changing it, and changing it back is more tricky than necessary, if possible it should be avoided. If the user might ctrl+c the script, open a new pane in tmux, or simply because of an internal bug, the user might find himself in the wrong place.

Many commands that accept filenames normally work with paths and those that do not might have a separate option for specifying the working directory.

For example, in git, instead of writing

it is possible to write git -C "$path" …​.

or, if -C is not available because your git version is too old, it is always possible to create a subshell

Bash offers additional features compared to other POSIX-compliant shells, but most of them do not add as much value as for the interactive shell. The common denominator for all shells on UNIX systems should be the POSIX specification. The more a script sticks to it, the easier it is to use with different shells (for example also when copy-pasting commands), and with different versions of the same shell. It might also make your scripts more efficient

On Windows, it’s a different story since the syntax of PowerShell, cmd and the POSIX specification are completely different. It’s unfortunate, but we can do nothing about it.

If you are unsure if a feature works on a POSIX-compliant shell and are working on a Debian-based system (or Cygwin), you could always try dash` (it might be installed by default on other systems too). You can use dash -n to check that a script will run with dash without actually executing it. It is not a perfect test but should be good enough for many purposes.

Of course, if you are already using bash and need some feature it adds (normally it should not be the case), just use it, unless you already know your script needs to be compatible with other shells too.

Not using some feature of bash might mean using an external tool, which might not be always available!

For example, the parameter expansion "${parameter/pattern/string}" is a bashism. To replace it, it is possible to use external tools like sed, grep, and/or awk. Those are all external dependencies. Granted, they should be on most systems. Those were probably not necessary before, so it is not obvious which dependency, the bash shell or those tools, is less of a burden.

Normally my shell scripts just invoke external tools, eventually, manipulate the outputs and call other tools. There is the hidden assumption that many "common" (mileage of common might vary) tools, like ls, find, …​ are available.

Sometimes a script needs to execute some tool that is normally not available, and the absence of such a tool is not necessarily an error. It might be an optional action or an action that can be performed with another tool, thus we want to know if at least one of those is available.

For example, how can we check for example if objdump is available without executing it?

command should cover most use cases:

The nice thing about command is that it works with aliases and functions too, so it is handy to use in .bashrc too.

Of course, it does not solve the problem if there exists a different version of the program that accepts different flags, maybe also depending on the platform or vendor of the tool. In that case, the only option might be to execute the tool and see if the flag is accepted. Consider installing the most recent and portable version, it might avoid a lot of headaches.

When the script gets too complex, change the scripting language.

sh is a very nice glue language; like all the others it has a lot of issues, and most of all, it is not very structured. It’s superb for trying something out, calling, and putting together a couple of tools, but writing complex logic and error handling is more difficult compared to most other languages.

Because nearly all distributions have Python (hopefully python3 in the meantime) installed, this is most of the time my first choice when "upgrading" a script. While I do not like Python that much, and of course it has its issues too, it makes it possible to write more structured code and still feels like a shell language. It is more verbose when calling external tools, but probably some of those tools can be replaced with library functions, this makes your code simpler, more portable, and easier to maintain.

In Unix-like systems, executing a Python script or a shell script is the same action. Through the shebang the system knows how to interpret the file.

Thus we can upgrade the script from one language to another without even changing the file name. Unfortunately, this is not possible on Windows, but we can always replace the old script with a new one that calls, for example, the Python script. This way it should be possible to avoid some transition issues if those scripts are deployed on multiple machines.