Useful bash commands (2)

File permissions

Each file and directory has a set of permission flags associated with it. You have seen them already as a sequence of ten characters in the first column of ls -l output:

$ ls -l foo
-rw-r--r--  1 dg  staff  6 23 Oct 12:00 foo

The first character indicates whether we are looking at a file (-) or a directory (d). Another frequently encountered first letter is l for links, which we’ll discuss later.

The next nine flags are grouped into three classes, called user, group and other. Within each class, the characters rwx show which permissions are set for that class. Each file belongs to exactly one user (here dg) and one group (here staff). In our example, the user dg is allowed to read and write to the file, while staff members and anyone else can only read the file. The execute permission (x) lets you run the file directly as a command.

On directories, the flags have a slightly different meaning: r only allows you to list the contents of the directory, but you need x to access them. w permits changes to the directory contents to be made.

chmod, chgrp, chown

chmod is the main tool to operate on file permissions. chown and chgrp can be used to change ownership of a file, but chown is often restricted to admin users. Any guess why?

Let’s play with the permissions to get some feel for how they work.

  1. Remove the read permission for yourself and try to cat a file.
  2. Remove the write permission for yourself and try to cat into a file.
  3. Try removing each one of rwx individually on a new directory and try to create, read, write to, rename or delete files inside the directory.
  4. Check if any files in your home directory are world writeable or world readable.

Command history

The command line looks quite primitive but has several features which allow you to work very efficiently. One of the most useful is the command history.

  • Run some commands from the last session, like ls, cd or find, then use the up and down arrows to avoid re-typing.
  • Go back in the command history and press Ctrl-o. What happens? If you have to repeat sequences of commands, this can be very handy!

What does history do?

$ man history

will tell you more than you’ll ever need to use.

What’s in the file ~/.bash_history? You can re-write history if you want.

Completion, wildcards and expansions

Before we look at tab completion and shell expansions we’ll need some files to work with. Change into the working directory for this course that you made last time and type:

$ touch fo foo foo1 foo2 foo3

List the directory to see what happened. touch creates empty files, or updates the timestamp on existing files.

Now try:

$ echo foo*

The shell expands * to produce a list of any files in the current directory whose names start with foo, followed by any number of characters. This expansion is done by the shell before the arguments are passed to the program. From echo’s point of view, the above line is equivalent to:

$ echo foo foo1 foo2 foo3

What happens if you type cat foo followed by the tab key twice?

Now try running:

$ ls -l fo*
$ touch fo?
$ ls -l fo*

From the modification times of the files, you can see that the ? character matches one single character. This meaning of the * and ? wildcards is known as glob expansion. “Globbing” is a simple version of much more powerful regular expressions.

Try typing:

$ ls
$ ls -l foo1
$ ls # -l foo1

What does the # do? This is useful when writing shell scripts. Now try:

$ echo foo#bar

Can you work out why the # doesn’t seem to work here?

Let’s try another kind of expansion: brace expansion. Run:

$ touch bar{10,20,30}

and list the files in the directory to see what happened. This can be very useful when copying or moving files with only small changes in the name:

$ mv fo{,o4}
$ mv bar{1,9}0

To better understand what the expansion is doing, try prefixing the line with echo:

$ echo mv fo{,o4}
$ echo mv bar{1,9}0

Remember why that works?

Now let’s clean up a bit:

$ rm foo*

Sometimes we don’t want expansion to happen. Double and single quote marks allow you to locally turn off bits of the shell’s syntax:

$ echo #foo
$ echo "#foo"
$ echo '#foo'

Single quotes are more restrictive than double quotes:

$ echo "hello!"
$ echo 'hello!'

What happened the first time? Can you use double quote symbols between double quotes? How about between single quotes? Can you get the terminal to echo the phrase "I'm not possible!" with all its quotes?

Actually, you can type double quotes inside double quotes, using a backslash escape character. You need escapes in any system with text delimiters where you sometimes need to show the delimiter itself. The escape character may differ, though.

  • How do you print a backslash itself using double quotes?
  • Using single quotes? Why does this mean you can never print out a single quote from between other single quotes, even with a backslash escape?

Environment variables and paths

The command line is not just manipulating files: the shell has an internal state determined by the values of environment variables. They can influence the way that programs run under the shell.

To see what variables are set in your environment, run:

$ env

or:

$ env | less -S

if you want to scroll around the output a bit.

You’ll notice that environment variables are usually given ALL-CAPITAL names. This is just a convention: if you define your own variables you can use any convention you like, although breaking conventions for no good reason is most likely to cause trouble later.

One of the most important environment variables is PATH. This is a list of file-system absolute paths (i.e. paths that start with the root location /), separated by colons.

You can have a look at your PATH variable either by scanning or grep-ing the output of env. Or, more instructively, by using variable expansion:

$ echo $PATH

Try printing out the values of the shell variables PWD, HOME, USER and SHELL this way.

Variable expansion is very useful: the $ sign prefixing the variable name means replace this variable name with its associated value. You can also use variable expansions inside double-quoted text strings quite neatly:

$ echo "My name is $USER"

So what is PATH used for? It is the list of directories that the shell searches for the commands you type. If you have two executable command files in separate directories listed in PATH, the one in the first listed directory will be run when you call that command name.

How would you add directories to your existing path?

To find out where e.g. the ls command is located, use:

$ which ls

Where is that directory in your PATH? Try with some other commands you’ve been using.

Note

Why shouldn’t you usually put . (the current directory) into your PATH, and definitely not early on?

What happens if two consecutive colons (::) are in your PATH?

To assign a value to a variable, we use an = sign:

$ MYFOO=bar
$ echo $MYFOO
bar

Note that the lack of spaces around the = sign is important — this is a bash syntax quirk that often catches people out. Also note that the dollar sign is not used when assigning to the variable name.

You can of course reference variable values in assignments, too:

$ MYBAR=foo${MYFOO}baz
$ echo $MYBAR
foobarbaz

What did the braces in ${MYFOO} do? What would happen if you didn’t use them?

The export command can be put in front of a variable assignment to make sure the variable is passed on to child processes. Here’s a demo:

$ foo=bar
$ bash   # open a new bash shell as a child of the previous one
$ echo $foo   # returns empty

Here foo is not a variable in the child process. Now press Ctrl-d to exit the child shell and try again with export:

$ export foo=bar
$ bash
$ echo $foo
bar

Does it work now?

Unix processes

Process control

Let’s see the processes that are running on your system:

$ ps -aux

As usual, look at man for the option switches.

You can see how the processes are forked from their parents by adding the f option to our ps command:

$ ps -axf | less -S

which will show a nice tree.

Try using grep via a pipe to get information about a particular process name, note that the grep itself will often match as well. Newer systems may have a command called pgrep, which is neater.

A handy interactive tool for looking at processes is top; you’ll end up using it a lot to work out what’s burning all your CPU or eating all your memory! top will show a regularly updating process list, you get back to the terminal by hitting q.

Processes can either be in the background or foreground. If the latter, the shell will not prompt again until they have finished: try running gedit and then re-focus the shell window. Your typing now isn’t registering as shell commands, it’s just appearing in the window while the editor sits there doing nothing. This is where the run command in the background syntax comes in.

Close gedit with a click (or highlight the terminal and press Ctrl-c to kill the foreground process), then try it again with:

$ gedit &

The terminal might get some messages written to it by gedit, but you should still be able to run commands: gedit is in the background. If you forget to put the & on the end, you can pause the foreground process with Ctrl-z, and then put it into the background with bg. The corresponding command fg will bring the most recent backgrounded command into the foreground. If a foreground process locks up so badly that you can’t even background it, you open another terminal and use a combination of ps and the kill or pkill commands to kill the offending process or its parent.

More powerful commands

The commands in this section are more command line applications with many diverse features, instead of just single-purpose tools. Still, they all have an option of reading from standard input and output, and can therefore be used in pipes.

ssh, scp

ssh allows secure remote login to other machines you have accounts on. To get an interactive login, use:

$ ssh your_username@hostname

Add the option -X to activate forwarding of graphical displays.

Alternatively, you can run commands remotely with:

$ ssh username@hostname command arguments

To copy files from remote locations, use:

$ scp filename filename

For a remote file, you can replace filename with username@hostname:filename.

A more advanced file copying program is rsync. This can save a lot of time if only a few files have changed, by only copying the ones which are different and by compressing them automatically. It uses ssh to do its network copying, but you may have to set RSYNC_RSH=ssh on older systems.

ssh keys (for reference, you can skip this for now)

An alternative to password authentication are ssh keys. To generate a set, use:

ssh-keygen -t dsa   # and just hit "enter" at the first question

This creates a directory .ssh which contains a private key id_dsa and a public key id_dsa.pub. On the remote system that you want to log in to, the public key should now be appended to .ssh/authorized_keys. You can either use ssh-send-id to do that, or use scp to copy the key to the other system by hand.

Warning

The private key should never be disclosed or made redable by anyone else!

At the next login to a machine that has your public key in the list of authorized keys, you’ll be asked for the key passphrase instead of the remote password. Try replicating this, then read on.

So far we haven’t gained any comfort: instead of the password you now have to type the key passphrase each time. But this part can be automated by using the ssh-agent service. On a graphical desktop it should already be running. Check with:

$ ps x | grep ssh-agent

If not, run it by hand for a child shell:

$ exec ssh-agent bash

You add a key to the agent with ssh-add. This will ask you for the passphrase only once. Any ssh or scp activity will now use the key, and will log you in to the remote machine without any additional requests. Using ssh keys will make remote version control operations especially convenient.

If you have trouble getting SSH keys to work, a common problem is file permissions: your home directory, ~/.ssh directory and the files within should not be group writeable. Also, the key files should be only readable by the owner. If these conditions aren’t met, chances are that SSH will just ask you for a password rather than explain what went wrong!

tar, gzip, bzip2

A convenient way of packaging directories is tar. Try:

$ tar cvf test.tar SomeDirectory

This will create a tarball containing a copy of all the files inside SomeDirectory. The option c stands for create, v is verbose and f is followed by the filename of the tarball. By convention, this should end in .tar.

To extract the files again somewhere else, run:

$ cd tmp
$ tar xvf /your/own/path/to/test.tar

tar tvf lists the files in a tarball.

gzip and bzip2 are two programs that can perform lossless compression on files to save space. Compare the size of 35.txt before and after running:

$ bzip2 35.txt

or:

$ gzip 35.txt

As you can see, by default the programs append .gz or .bz2 and erase the original file. To uncompress a file, run:

$ bunzip2 35.txt.bz2

or:

$ gunzip 35.txt.gz

tar can compress its tarball automatically, without needing to run gzip afterwards. To do this, add the option z for gzip or j for bzip2 to the tar command line. To avoid confusion, you should provide the right file endings .tar.bz2 or .tar.gz.

A piped use of tar is the following useful line. What does it do?:

$ ssh name@location "tar cj SomeDirectory" | tar xj

When dealing with thousands of files inside the directory, this is much faster than scp -r. If you need do such a transfer regularly, you’re better off with rsync, though.

What does the next line do?:

$ tar cj SomeDirectory | ssh name@host "cat > archive.tar.bz2"

Editing text

Text editors belong to the basic toolkit in any operating system. The choice in Linux is correspondingly large. On a typical system, you’ll find nano for quick edits, emacs or vi for larger projects. gedit or kate are more integrated into the graphical OS. Many more editors exist. Any of them do the basic job of editing a text file very well, and should have syntax highlighting.

To find an editor you like, try them out for a while. Here’s a large-ish file from project Gutenberg: [H.G. Wells, The Time Machine]. Right-click to copy the link’s URL and use wget to download it from the shell.

Open the file in the editors you want to try out with someeditorname 35.txt & (do you remember what the & does?) and try something like

  • Find the first occurrence of of course in the text.
  • Replace all occurrences of time with foo.
  • Mark a section of the text, cut it and move it somewhere else.