Useful bash commands (2)
========================

.. highlight:: console

.. role:: bash(code)
   :language: bash
   :class: highlight

File permissions
----------------

Each file and directory has a set of :dfn:`permission flags`
associated with it. You have seen them already as a sequence of ten
characters in the first column of :bash:`ls -l` output::

	$ ls -l foo
	-rw-r--r--  1 dg  staff  6 23 Oct 12:00 foo

The first character indicates whether we are looking at a file
(:kbd:`-`) or a directory (:kbd:`d`). Another frequently encountered
first letter is :kbd:`l` for links, which we'll discuss later.
 
The next nine flags are grouped into three :dfn:`classes`, called
:dfn:`user`, :dfn:`group` and :dfn:`other`.  Within each class, the
characters :kbd:`rwx` show which permissions are set for that
class. Each file belongs to exactly one user (here :kbd:`dg`) and one
group (here :kbd:`staff`). In our example, the user :kbd:`dg` is
allowed to read and write to the file, while :kbd:`staff` members and
anyone else can only read the file. The execute permission (:kbd:`x`)
lets you run the file directly as a command. 

On directories, the flags have a slightly different meaning: :kbd:`r`
only allows you to list the contents of the directory, but you need
:kbd:`x` to access them. :kbd:`w` permits changes to the directory
contents to be made.

chmod, chgrp, chown
~~~~~~~~~~~~~~~~~~~

:command:`chmod` is the main tool to operate on file
permissions. :command:`chown` and :command:`chgrp` can be used to
change ownership of a file, but :command:`chown` is often restricted to
admin users. Any guess why?

Let's play with the permissions to get some feel for how they
work.

#. Remove the read permission for yourself and try to :bash:`cat` a file.

#. Remove the write permission for yourself and try to :bash:`cat`
   *into* a file.

#. Try removing each one of :kbd:`rwx` individually on a new directory
   and try to create, read, write to, rename or delete files inside the
   directory.

#. Check if any files in your home directory are world writeable or world
   readable. 


..
  \timefornotes{Shell convenience features}{
  \begin{itemize}
  \item History
  \item Aliases
  \item Brace and variable expansions
  \item Environment variables
  \end{itemize}
  }

Command history
---------------

The command line looks quite primitive but has several features which
allow you to work very efficiently. One of the most useful is the
:dfn:`command history`.  

- Run some commands from the last session, like :bash:`ls`, :bash:`cd`
  or :bash:`find`, then use the up and down arrows to avoid
  re-typing. 

- Go back in the command history and press :kbd:`Ctrl-o`. What
  happens? If you have to repeat sequences of commands, this can be
  very handy!

..
  %% Try pressing \kbd{Ctrl-r} and typing a few consecutive characters
  from a co  mmand
  %% you've uased recently: what happens? Try pressing \kbd{Ctrl-r} again a few times.
  %% \footnote{\kbd{Ctrl-s} does the same thing, but going forwards. It tends to not
  %%   work by default, though: you have to tweak your shell configuration using \inp{stty -ixon}}
  %% Try typing \inp{!}, followed by the first few letters of a recent command.

What does :command:`history` do? ::

  $ man history

will tell you more than you'll ever need to use. 

What's in the file :file:`~/.bash_history`? You can re-write history
if you want.


Completion, wildcards and expansions
------------------------------------

Before we look at :dfn:`tab completion` and :dfn:`shell expansions`
we'll need some files to work with. Change into the working directory
for this course that you made last time and type::

  $ touch fo foo foo1 foo2 foo3

List the directory to see what happened. :command:`touch` creates
empty files, or updates the timestamp on existing files. 

Now try::

  $ echo foo*

The shell :dfn:`expands` :kbd:`*` to produce a list of any files in
the current directory whose names start with :file:`foo`, followed by
any number of characters. This expansion is done by the shell *before*
the arguments are passed to the program. From :command:`echo`'s point
of view, the above line is equivalent to::

  $ echo foo foo1 foo2 foo3

What happens if you type :kbd:`cat foo` followed by the tab key twice?

Now try running::

  $ ls -l fo*
  $ touch fo?
  $ ls -l fo*

From the modification times of the files, you can see that the
:kbd:`?` character matches one single character. This meaning of the
:kbd:`*` and :kbd:`?` :dfn:`wildcards` is known as :dfn:`glob
expansion`. "Globbing" is a simple version of much more powerful
:dfn:`regular expressions`.

Try typing::

  $ ls
  $ ls -l foo1
  $ ls # -l foo1

What does the :kbd:`#` do? This is useful when writing :dfn:`shell
scripts`. Now try::

  $ echo foo#bar

Can you work out why the :kbd:`#` doesn't seem to work here?

Let's try another kind of expansion: :dfn:`brace expansion`. Run::

  $ touch bar{10,20,30}

and list the files in the directory to see what happened. This can be very
useful when copying or moving files with only small changes in the
name::

  $ mv fo{,o4}
  $ mv bar{1,9}0

To better understand what the expansion is doing, try prefixing the
line with :command:`echo`::

  $ echo mv fo{,o4}
  $ echo mv bar{1,9}0

Remember why that works?

Now let's clean up a bit::

  $ rm foo*

.. warning: 

   You should always be careful when using :command:`rm`,
   especially when using the :option:`-r` and :option:`-f` options. The
   shell is powerful, but if you're not careful it will happily delete all
   your files.

Sometimes we don't *want* expansion to happen. Double and single quote
marks allow you to locally turn off bits of the shell's syntax::

  $ echo #foo
  $ echo "#foo"
  $ echo '#foo'

Single quotes are more restrictive than double quotes::

  $ echo "hello!"
  $ echo 'hello!'

What happened the first time? Can you use double quote symbols between double
quotes? How about between single quotes? Can you get the terminal to echo the
phrase :kbd:`"I'm not possible!"` with all its quotes?

Actually, you *can* type double quotes inside double quotes, using a
backslash :dfn:`escape character`. You need escapes in any system with
text delimiters where you sometimes need to show the delimiter
itself. The escape character may differ, though. 

- How do you print a backslash itself using double quotes?  
- Using single quotes? Why does this mean you can
  never print out a single quote from between other single quotes, even
  with a backslash escape?

Environment variables and paths
-------------------------------

The command line is not just manipulating files: the shell has an
internal state determined by the values of :dfn:`environment variables`. They
can influence the way that programs run under the shell.

To see what variables are set in *your* environment, run::

  $ env

or::

  $ env | less -S

if you want to scroll around the output a bit. 

You'll notice that environment variables are usually given ALL-CAPITAL names. 
This is just a convention: if you define your own variables you can use any 
convention you like, although breaking conventions for no good reason is most
likely to cause trouble later.

One of the most important environment variables is
:envvar:`PATH`. This is a list of file-system absolute paths
(i.e. paths that start with the root location :file:`/`), separated by
colons. 

You can have a look at your PATH variable either by scanning or
:command:`grep`-ing the output of :command:`env`. Or, more instructively, by
using :dfn:`variable expansion`::

  $ echo $PATH

Try printing out the values of the shell variables :envvar:`PWD`,
:envvar:`HOME`, :envvar:`USER` and :envvar:`SHELL` this way.

Variable expansion is very useful: the :kbd:`$` sign prefixing the
variable name means *replace this variable name with its associated
value*. You can also use variable expansions inside double-quoted
text strings quite neatly::

  $ echo "My name is $USER"

So what is :envvar:`PATH` used for? It is the list of directories that
the shell searches for the commands you type. If you have two
executable command files in separate directories listed in
:envvar:`PATH`, the one in the first listed directory will be run when
you call that command name.  

How would you add directories to your existing path?

To find out where e.g. the :command:`ls` command is located, use::

  $ which ls

Where is that directory in your :envvar:`PATH`? Try with some other
commands you've been using. 

.. note::
  Why shouldn't you usually put :kbd:`.` (the current directory)
  into your :envvar:`PATH`, and definitely not early on? 

  What happens if two consecutive colons (:kbd:`::`) are in your
  :envvar:`PATH`? 

..
  \inp{cd /home/hudson/staff/dph1dg1/never-put-dot-in-path}\\
  \inp{ls}\\
  to see if you're OK.

To assign a value to a variable, we use an :kbd:`=` sign::

  $ MYFOO=bar
  $ echo $MYFOO
  bar

Note that the lack of spaces around the :kbd:`=` sign *is* important
--- this is a bash syntax quirk that often catches people
out. Also note that the dollar sign is *not* used when assigning to the
variable name.

You can of course reference variable values in assignments, too::

  $ MYBAR=foo${MYFOO}baz
  $ echo $MYBAR
  foobarbaz

What did the braces in :bash:`${MYFOO}` do? What would happen if you
didn't use them?

The :command:`export` command can be put in front of a variable
assignment to make sure the variable is passed on to child processes. 
Here's a demo::

  $ foo=bar
  $ bash   # open a new bash shell as a child of the previous one
  $ echo $foo   # returns empty

Here :envvar:`foo` is not a variable in the child process. 
Now press :kbd:`Ctrl-d` to exit the child shell and try
again with :command:`export`::

  $ export foo=bar
  $ bash
  $ echo $foo
  bar

Does it work now?


Unix processes
--------------

..
  \begin{itemize}
  \item Tree structure again --- cut the root to kill the branches
  \item Every branch (process) has an ID number
  \item Signals: :kbd:`kill} and :kbd:`pkill}
  \item Detaching
  \item Environment propagation: :kbd:`export}
  \item A process can't affect the environment of its parent
  \end{itemize}

Process control
~~~~~~~~~~~~~~~

..
  You should now know something about how Unix processes work. 

Let's see the processes that are running on your system::

  $ ps -aux

As usual, look at :command:`man` for the option switches.

You can see how the processes are forked from their parents by adding the :kbd:`f` option to our :command:`ps` command::

  $ ps -axf | less -S

which will show a nice tree. 

Try using :command:`grep` via a pipe to get information about a particular process name, note that the grep itself will often match as well. Newer systems may have a command called :command:`pgrep`, which is neater. 

A handy interactive tool for looking at processes is :command:`top`;
you'll end up using it a lot to work out what's burning all your CPU or eating
all your memory! :command:`top` will show a regularly updating process list, you get back to the terminal by hitting :kbd:`q`. 

Processes can either be in the :dfn:`background` or :dfn:`foreground`. If the latter, the shell will not prompt again until they have finished: try running :bash:`gedit` and then re-focus the shell window. Your typing now isn't registering as shell commands, it's just appearing in the window while the editor sits there doing nothing. This is where the *run command in the background* syntax comes in.

Close gedit with a click (or highlight the terminal and press :kbd:`Ctrl-c` to
kill the foreground process), then try it again with::

  $ gedit &

The terminal might get some messages written to it by gedit, but you should
still be able to run commands: gedit is in the background. If you forget to put
the :kbd:`&` on the end, you can pause the foreground process with
:kbd:`Ctrl-z`, and then put it into the background with :command:`bg`. The
corresponding command :command:`fg` will bring the most recent backgrounded command into the foreground. If a foreground process locks up so badly that you can't even background it, you open another terminal and use a combination of :command:`ps` and the :command:`kill` or :command:`pkill` commands to kill the offending process or its parent.


More powerful commands
----------------------

The commands in this section are more command line
*applications* with many diverse features, instead of just single-purpose
tools. Still, they all have an option of reading from standard input
and output, and can therefore be used in pipes.

ssh, scp
~~~~~~~~

:command:`ssh` allows secure remote login to other machines you have accounts on. To get an interactive login, use::

  $ ssh your_username@hostname

Add the option :option:`-X` to activate forwarding of graphical displays.

Alternatively, you can run commands remotely with::

  $ ssh username@hostname command arguments

..
  Try to log in to the guest accounts on the whiteboard
  \inp{ssh username@hostname cat /proc/cpuinfo}

To copy files from remote locations, use::

  $ scp filename filename

For a remote file, you can replace :bash:`filename` with 
:bash:`username@hostname:filename`. 

..
  An example is::
  \inp{scp foo dph...@s5.phyip3.dur.ac.uk:/scratch/foo}\\
  (unlike the home directories, the :bash:`/scratch} directories are local
  to each machine) 

A more advanced file copying program is :command:`rsync`. This can save a lot of time if only a few files have changed, by only copying the ones which are
different and by compressing them automatically. It uses :command:`ssh` to do its network copying, but you may have to set :envvar:`RSYNC_RSH=ssh` on older systems.

 
ssh keys (for reference, you can skip this for now)
___________________________________________________

An alternative to password authentication are :dfn:`ssh keys`. To
generate a set, use::

  ssh-keygen -t dsa   # and just hit "enter" at the first question

This creates a directory :file:`.ssh` which contains a :dfn:`private key`
:file:`id_dsa` and a :dfn:`public key` :file:`id_dsa.pub`. On the remote 
system that you want to log in to, the public key should now be appended 
to :file:`.ssh/authorized_keys`. You can either use :command:`ssh-send-id` 
to do that, or use :command:`scp` to copy the key to the other system by hand.

.. warning::

  The private key should *never* be disclosed or made redable by anyone else!

At the next login to a machine that has your public key in
the list of authorized keys, you'll be asked for the key passphrase
instead of the remote password. Try replicating this, then read on.

So far we haven't gained any comfort: instead of the password you now have to type the key passphrase each time. But this part can be automated by using the
:command:`ssh-agent` service. On a graphical desktop it should
already be running. Check with::

  $ ps x | grep ssh-agent

If not, run it by hand for a child shell::

  $ exec ssh-agent bash

You add a key to the agent with :command:`ssh-add`. This will ask you for
the passphrase only once. Any :command:`ssh` or :command:`scp` activity will now
use the key, and will log you in to the remote machine without any
additional requests. Using ssh keys will make remote version control operations
especially convenient.

If you have trouble getting SSH keys to work, a
common problem is file permissions: your home directory, :kbd:`~/.ssh`
directory and the files within should not be group writeable. Also, the
key files should be only readable by the owner. If these conditions aren't
met, chances are that SSH will just ask you for a password rather than 
explain what went wrong!

.. 
  TODO
  mention .ssh/config

tar, gzip, bzip2
~~~~~~~~~~~~~~~~

A convenient way of packaging directories is :command:`tar`. Try::

  $ tar cvf test.tar SomeDirectory

This will create a :dfn:`tarball` containing a copy of all the files
inside :file:`SomeDirectory`. The option :option:`c` stands for
*create*, :option:`v` is *verbose* and :option:`f` is followed
by the filename of the tarball. By convention, this should end in :file:`.tar`.

To extract the files again somewhere else, run::

  $ cd tmp
  $ tar xvf /your/own/path/to/test.tar

:bash:`tar tvf` lists the files in a tarball.

:command:`gzip` and :command:`bzip2` are two programs that can perform
*lossless compression* on files to save space. Compare the size
of :file:`35.txt` before and after running::

  $ bzip2 35.txt

or::

  $ gzip 35.txt

As you can see, by default the programs append :file:`.gz`
or :file:`.bz2` and erase the original file. To uncompress a file, run::

  $ bunzip2 35.txt.bz2

or::

  $ gunzip 35.txt.gz

:command:`tar` can compress its tarball automatically, without needing to
run :command:`gzip` afterwards. To do this, add the option :option:`z` for
:command:`gzip` or :option:`j` for :command:`bzip2` to the :command:`tar` command line. To avoid confusion, you should provide the right file endings
:file:`.tar.bz2` or :file:`.tar.gz`. 

A piped use of :command:`tar` is the following useful line. What does it
do?::

  $ ssh name@location "tar cj SomeDirectory" | tar xj

When dealing with thousands of files inside the directory, this
is *much* faster than :bash:`scp -r`. If you need do such a transfer regularly, you're better off with :command:`rsync`, though.

What does the next line do?::

  $ tar cj SomeDirectory | ssh name@host "cat > archive.tar.bz2"

mail, wget, elinks
~~~~~~~~~~~~~~~~~~

To wrap up, these are some command-line alternatives for graphical
programs: :command:`mail` is a mail program (most useful for sending), :command:`wget` is a command-line file downloader and :command:`elinks` a web browser.

Try emailing the output of :bash:`ls -l` to yourself using a command line pipe::

  $ ls -la | mail -s "Testing mail pipe" my.email@example.com


Editing text
~~~~~~~~~~~~

Text editors belong to the basic toolkit in any operating
system. The choice in Linux is correspondingly large. On a typical system,
you'll find :command:`nano` for quick edits, :command:`emacs` or
:command:`vi` for larger projects. :command:`gedit` or :command:`kate` are more integrated into the graphical OS. Many more editors exist. Any of them do the basic job of editing a text file very well, and should have :dfn:`syntax highlighting`.

To find an editor you like, try them out for a while. Here's a large-ish file from project Gutenberg: :download:`[H.G. Wells, The Time Machine] <_static/35.txt>`. Right-click to copy the link's URL and use :command:`wget` to download it from the shell.

Open the file in the editors you want to try out with :bash:`someeditorname 35.txt &` (do you remember what the :kbd:`&` does?) and try something like

- Find the first occurrence of *of course* in the text.
- Replace all occurrences of *time* with *foo*.
- Mark a section of the text, cut it and move it somewhere else.