[logo] [title]

High Performance Computing

Introduction to Linux

Getting started

Before continuing to the exercises below, ensure that you are logged into a linux system and that you have an open terminal window. To start a terminal window from a Gnome desktop as in Ubuntu, click Applications - Accessories - Terminal to start ...

The following assumes you are using the bash shell, which is the default.

The bash shell and editors

Now you are ready to begin this session. Start by entering the command 'pwd'

[user@comp mijp1]$ pwd

This prints the Present Working Directory to the screen. This will be /home followed by your username. This is your home directory in which you can create and manipulate files. All user directories are stored under /home. Other directories in the Linux file-system are /etc where system-wide configuration files are stored and /bin where system programs are stored, plus many others.

To list the files which are currently stored in your home directory use the 'ls' command.

[user@comp mijp1]$ ls

Those of you new to linux will notice that you have very few (if any) files. To learn more about the ls command type.

[user@comp mijp1]$ man ls

This brings up the manual page for ls. Use the arrow keys to scroll through the data and press q to quit when done.

Now let's make a directory we can store files in for this tutorial using the mkdir command.

[user@comp mijp1]$ mkdir intro
[user@comp mijp1]$ ls

The ls command now shows that the directory has been created. Now change into this directory using the cd command, and check that we have done so using pwd as above.

[user@comp mijp1]$ cd intro
[user@comp intro]$ pwd

Note that the prompt has changed to reflect the present working directory. Now lets create some new directories within 'intro', one to store FORTRAN code and one to store c code.

[user@comp intro]$ mkdir c
[user@comp intro]$ mkdir fortran

You can check these new directories now exist using 'ls' as before. I've prepared some example code for you to play with - right click these files in your browser and do a "save link" to put into the appropriate directories:

Now let's have a look at the c program we've downloaded. First change into the c directory.

[user@comp intro]$ cd c

and edit the file prime.c. There are several editors you can use to do this. To use the kate editor use the command

[user@comp c]$ kate prime.c &

This launches the kate program. The & on the end of the command runs kate in the background so that we can keep using our terminal window without closing kate.

To learn more about kate you can use the help menu it provides or type

[user@comp c]$ man kate

into a terminal window.

You may find you prefer another editor such as emacs

[user@comp c]$ emacs prime.c &

or gedit which is part of the GNOME desktop suite of programs and utilities.

[user@comp c]$ gedit prime.c &

Feel free to play around with each editor and decide which one is for you. In particular, try to find the syntax highlighting option if this is not turned on by default. Many people find highlighted code easier to read and edit. Note that to learn more about emacs you can type

[user@comp c]$ man emacs

return to top

Compiling and running C code

Let's compile prime.c into a program using the GNU C compiler.

[user@comp c]$ gcc prime.c

Nothing obvious has happened yet. Try listing the contents of the present working directory.

[user@comp c]$ ls

See anything unexpected? The compiler has created a new file called a.out. This is an executable file which we can invoke to run the code we compiled. To do this we just type the path to this file using the shorthand './' for the present working directory.

[user@comp c]$ ./a.out

Run the program a few times and convince yourself that it works. This is a very simple program which calculates if a number you provide (you must press return after you enter the number) is prime.

We can interact with this program in different ways. For example using the echo command. Try the command

[user@comp c]$ echo Hello World

which should type 'Hello World' into our terminal window. We can feed the output of the echo command into our program using a 'pipe' which is represented by the solidus symbol.

[user@comp c]$ echo 5 | ./a.out

Now a.out isn't a very good name for a program. Let's recompile and call it something different using the -o (lower case) option to gcc.

[user@comp c]$ gcc prime.c -o prime.exe

A quick ls command will show that we have a new file called prime.exe. Let's give it a big number to work on, and time how long it takes using the time command.

[user@comp c]$ time echo 1000000097 | ./prime.exe

We can do better than this by letting the compiler optimise the code (at say level 3) using the -O (upper case) option to gcc.

[user@comp c]$ gcc prime.c -O3 -o fastprime.exe

Repeat the time test with the optimised version.

[user@comp c]$ time echo 1000000097 | ./fastprime.exe

This should be faster, but the actual speedup depends on the code, the computer and the size of the problem. To be more precise about the timings we should run the same task multiple times to get an average time.

To learn more about the various options to gcc type

[user@comp c]$ man gcc

The bash shell can do more than copy files and invoke programs. Let's get really flashy and make it do a loop.

[user@comp c]$ for i in {1..99}
> do
> echo $i | ./fastprime.exe
> done

This runs our prime number program for all integers from 1 to 99.

return to top

Compiling and running Fortran Code

Now change into the fortran directory we created earlier

[user@comp c]$ cd ~/intro/fortran
[user@comp fortran]$ ls

We have used the shorthand ~ instead of the full path to our home directory.

There are three fortran source code files in this directory, plus a README file which you can examine if you wish. Open the source files in your favourite editor and explore them a little.

Now lets compile the three source files into a program using the GNU Fortran 95 compiler, with some optimisation.

[user@comp fortran]$ gfortran -O3 constants.f90 gravity.f90 earthmoon.f90 -o earthmoon.exe

(all on one line). The order in which the source files are specified is important. earthmoon.f90 uses functions and subroutines in gravity.f90, which in turn uses constants.f90

Notice that some of the options for gfortran match those for gcc. Other GNU compilers include g77 for fortran 77 and g++ for C++. Now run the program

[user@comp fortran]$ ./earthmoon.exe

This program prints a lot of information to the screen which should be fairly self explanatory. As you might have guessed, this program performs a short simulation of the Earth-Moon gravitational system. At each time stop the three components of the position vectors (in Cartesians) are output to the screen.

We can redirect the output into a file using the > operator

[user@comp fortran]$ ./earthmoon.exe > output.dat
[user@comp fortran]$ ls

The ls command should now show a new file called output.dat. To show the contents of this file you can edit it with your favourite editor, or using the more or less commands.

[user@comp fortran]$ more output.dat
[user@comp fortran]$ less output.dat

If using less, you can scroll with the arrow keys and exit with the 'q' key.

The grep command is very useful.

[user@comp fortran]$ grep Moon output.dat

This only prints the lines in output.dat which contain the word 'Moon' (case sensitive). We can pipe the output of grep into another program called awk and tell it to just print the 6th and 8th columns.

[user@comp fortran]$ grep Moon output.dat | awk '{print $6,$8}'

This prints just the x and z coordinates of the moon to screen. To learn more about grep and awk, use

[user@comp fortran]$ man grep
[user@comp fortran]$ man awk

respectively. Hopefully we are learning that the 'man' command is your friend. Use it, know it, love it.

Now redirect the moon coordinates into a file.

[user@comp fortran]$ grep Moon output.dat | awk '{print $6,$8}' > moon.dat

We can plot this file using xmgrace.

[user@comp fortran]$ xmgrace moon.dat &

Looks like a pretty nice orbit to me. Another plotting program is gnuplot

[user@comp fortran]$ gnuplot

we now have a new command prompt in which we can enter plot commands such as

gnuplot> p 'moon.dat'

Both xmgrace and gnuplot can be used to create publication quality figures. You might also want to look at the xfig program. Read the man pages and the online help to learn more. Now exit gnuplot

gnuplot> quit

Delete all our output using the rm (remove) command.

[user@comp fortran]$ rm *.dat
[user@comp fortran]$ ls

The 'ls' command should confirm that the files output.dat and moon.dat are gone.

Launch your favourite editor and type in the following.

#!/bin/bash

# this is a comment
./earthmoon.exe > output.dat
grep Moon output.dat | awk '{print $6,$8}' > moon.dat

Save this file as moonscript.sh in your fortran directory. The first line tells the system to interpret the commands in this file using the bash shell. This is an example of a shell script. We can make the moonscript.sh file executable using

[user@comp fortran]$ chmod u+x moonscript.sh

We can now run this script using

[user@comp fortran]$ ./moonscript.sh

A quick 'ls' command should confirm that the output.dat and moon.dat files have been recreated by the script.

If you want to know more about some of these core utilities, then you might like this simple guide to grep, awk and sed that I have prepared.

return to top

Exercises

1) A useful tool for finding out about the hardware on a machine is 'dmesg' which reports all that happened (including any errors) during boot-up and if you add/remove any hardware whilst the system is running:

[user@comp mijp1]$ dmesg

Linux also has a 'virtual filesystem' for tracking what is going on within the machine, which you can use to see more about the CPU and the memory etc:

[user@comp mijp1]$ cat /proc/cpuinfo
[user@comp mijp1]$ cat /proc/meminfo

This produces a lot of information. Try using 'grep' to locate information on the CPU in particular on any linux machine.

2) Spy on what is happening on the machine you are logged into!

[user@comp mijp1]$ top

This displays the list of running processes on the machine we are logged into. To see information on the processes of just one user (e.g me) use the ps command.

[user@comp mijp1]$ ps -u mijp1

You can see who is logged into the current machine using the 'finger' program.

[user@comp mijp1]$ finger

This gives information on who is logged in, from which machine, when they logged in and how long their terminal session has been idle.

You can also finger individual users. If a user has created a file called '.plan' in their home directory, fingering that user will display the contents of that file. Try using the finger command to show my (mijp1) plan.

Finally you can use the 'last' command to display a history of logins since the beginning of the month.

[user@comp mijp1]$ last

This gives a lot of information. By piping the output of last into another tool called 'head', we can display just say the first 10 entries.

[user@comp mijp1]$ last | head -n10

There is a similar command called 'tail' for displaying just the end of a file.

Use the man pages if you want any more information on any of these commands.

3) Groups and permissions. At some point you will be using a remote Linux system for assessed work which you may not want other users to have access to! Permissions to files can be set at three levels, owner group and other. Use the groups command to find out which group(s) you are a member of on any machine.

[user@comp mijp1]$ groups

On some systems, each user has his or her own private group, or you may be in the student group unless you work in a specific research group. Go into your home directory and type

[user@comp mijp1]$ ls -l

to display a list of your files and permissions. The first column contains 10 letters. The first letter indicates if the entry on this line is an ordinary file (-) a directory (d) or a link (l) to another file or directory.

The following three letters indicate the permissions on this file for its owner. These permissions are (r) for read (w) for write, and (x) for execute. Note that a directory needs execute privileges to be entered. The following three letters indicate the same three privileges for the owners group, and the final three give the privileges for all other users. For example, in my home directory I have a directory called lib

drwxr-xr-x 2 mijp1 staff 4096 Sep 22 20:15 lib

which has read and execute privileges for the owner (me) my group and all other users. The third and fourth columns indicate the username and group of the owner of the file (i.e. me).

To change the permissions on a file (you must be the owner) use the 'chmod' command. Create an empty file using the touch command.

[user@comp mijp1]$ touch myfile.txt

A quick 'ls' shows that the file has been created. As an example, let's add group read and write privileges and remove these privileges for other users.

[user@comp mijp1]$ chmod g+rw myfile.txt
[user@comp mijp1]$ chmod o-rw myfile.txt

Repeating the 'ls -l' command should confirm that these have been set correctly. To learn more about the chmod command use the man pages.

4) Text files. Linux comes with many useful commands for examining and manipulating text files. This is particularly useful for the programmer, as it makes more sense to generate results from your code and write them to a file, and then manipulate using the shell, rather than to try and do too much within your program. So in the 'earthmoon' example, we wrote out all the data and used various commands to get the specific data we needed for plotting, rather than write our own plotting routines!

Useful commands that you should be confident in using are:

[user@comp mijp1]$ cat
[user@comp mijp1]$ head
[user@comp mijp1]$ tail
[user@comp mijp1]$ more
[user@comp mijp1]$ less
[user@comp mijp1]$ grep
[user@comp mijp1]$ awk

return to top

Bash shell tips

When using the terminal window, the up and down arrows on the keyboard can be used to scroll through previous commands.

When entering the name of a program or file, press the tab key before you before you enter the full name. The shell will complete the command for you if it finds a unique match for what you have typed so far.

To repeat the previous command simply type an exclamation mark.

Phil Hasnip has produced a simple Linux tutorial and also a top-20 list of useful commands.

For many more little gems see this web site.

This tutorial has covered only a few Linux terminal commands. There are many more. If I've missed out anything you wanted to know then shout.

return to top