Lecture 02: Unix Familiarization and Command Line Tools
Table of Contents
1 Unix File System and Command Line Tools intro
1.1 File System Preliminaries
The file system is a way to organize data into files and directories for ease of access. A file system is hierarchal, which means that everything is arranged in order of rank, like your chain of command. We describe hierarchical structures like this as trees.
Consider the tree above. All trees are composed of three components,
root, sub-roots, and leaves. The root of the tree is the
top-ranked item, in the example above, this would be
foo. Sub-roots are roots of smaller trees, but are not at the top,
for example, bar is a sub-root. Leaves exists at the bottom, and
do not have any sub-ranked items.
We also describe items as having parents and children. What
makes an item a root is that it has no parent, like foo. A
sub-root, can be defined as an item that has a parent and children;
like bar, whose parent is foo and children are xyzzy and
baz. The leaves are items that only have parents and no children,
like garply and baz.
In a file system, we have terms for roots and sub roots. They are called folders or directories. A folder can contain folders and files, which is a child in the tree. A folder may also be a child in the tree if it contains no sub-folders or files.
A path in the file system hierarchy for a given file or folder
describes the parents all the way up to the root. For example, the
path of baz is /foo/bar/baz. We designate the parent-child
relationship using the forward slash.
A path also provides a way to uniquely identify a file. Two files
can have the same file name, but may not have the same path. For
example, baz exists twice in the file system above, but they have
two different paths: /foo/baz and /foo/bar/baz. Thus, they can
be distinguished, but if the two files had the same path, that is,
existed in the same directory, then they would not be
distinguishable and an inconsistency would occur in the file
system.
1.2 The UNIX File System
You are likely familiar with the Windows file system structure, which is organized into drives, like the "C" drive, and then items fall out from there. There can be many drives attached to the Windows file system, and you navigate under various drives by letter name.
On Unix, instead of having a "C" drive, everything begins at the root directory or /, and unlike windows, Unix uses forward slashes instead of backslash to designate different directories.
1.2.1 The Key Components of UNIX File System
The base UNIX file systems always has the same basic structure:
/: The root of the file system. All files and directories fall under this./usr: Stands for Unix System Resources. Contains system utilities./sbin: System binaries. Contains essential system administration programs that are generally run by the superuser (on Windows the superuser is called the Administrator).
/opt: Optional software. Third party software is usually installed here. It's kind of like the "Program Files" directory in Windows./etc: System configuration files. This is where things like the password file, and global system configuration files live./home: Contains the user home directories. Like the "Documents and Settings" folder in Windows./tmp: Temporary files. When the system reboots, these go away./kernel: The core operating system. Like the "Windows" folder in Windows./usr/lib: Contains precompiled libraries for use by everyone on the system. For instance, in this directory, you can find the file libstdc++.so.6, which is needed for C++ programs. (Remember linking from IC210?). This is a little bit like the folder containing all of the.DLLfiles in Windows.- Any directory that ends in
bin: Contains binary executable files or links to them.
1.2.2 Home Directories
Each user on the UNIX file system has a /home directory in the
/home folder.
- My username is aviv, then my home directory is
/home/avivon my VM and/home/scs/avivon a lab computer. - Your home directory will be some thing like
/home/m18XXXXXon your VM or something like/home/mids/m18XXXXon a lab computer. - The ~ (tilda) is short hand for your home directory.
- For example, if you use a path like
~/Downloadsyou are referring theDownloadsfolder in your home directory. The~is replaced by/home/m18XXXXautomatically.
- For example, if you use a path like
- You can also refer to someone else home directory via the
tilda,
~avivrefers to aviv's home directory, and~m18xxxxmay refer to your home directory.- For example,
~aviv/Dowloadsrefers to the downloads directory in useraviv's home directory and~m18xxxx/Downloadsrefers to the downloads directory in userm18xxxx's home directory.
- For example,
1.2.3 Unix Directory Paths
+--- Root Directory
|
V
/home/aviv/foo.txt
^ ^ ^
Sub-Directory ---'---' '-- Target
1.2.4 Parent and Current Directory
Every directory has two special sub-directory:
- . : ("dot") The current directory
- .. : ("dot-dot") The parent directory
Another way to interpret the dot and dot-dot is by replacing them with "current" and "parent". Consider the path below:
/home/aviv/../m175678
Reading it from left to right, you might say: "From the root, go to
home directory, then to the aviv directory, then to the parent
of aviv, then to m175678, the target." Here is another path:
/home/aviv/./././../aviv/./foo.txt
The dot is replaced with current folder, and by stringing then
together, it has no affect on the path. In the example above, all
the dots refer to the current directory, aviv. Following the rest
of the path, the dot-dot then refers to go to the parent directory,
home, but that is followed by traversing back to the aviv
directory. Then the dot means to stay put again, and then, finally,
the target, foo.txt.
1.3 The Shell Preliminaries
The shell or terminal is the primary user interface for interacting with the operating system through text input. You have already used various shell's previously, like the Windows command terminal as well as the Unix shell. You compile C++ programs by typing a command, and then the shell tells the OS to execute those commands.
A shell is just another program running on the OS, like any other
application. Most shells are the program, bash, which we will look
at in detail later. What makes a shell special is that it is a
program designed specifically to enable the user to launch other
programs. In many ways, it is the primary user interface of the
OS. Additionally, the shell and the OS on Unix provide a simple set
of command line tools that enable you to navigate the file system,
manipulate the file system by creating or deleting files/folders,
read and parse files on the system, and monitor current running
process and programs.
Today, we'll focus on the command line tools associated with the navigating and manipulating the file system. As we work through this class we will dive into the Unix system by emphasizing shell scripting and interacting with the Unix kernel through C system call API.
1.3.1 Current Working Directory
The shell has a notion of location, or current working directory or present working directory, that indicates where in the file system the shell is currently operating. When you first log into a computer, a shell is started for you set to your home directory as the current working directory. The shell can change the current working directory to view a different parts of the file system.
1.3.2 Navigating the file system
There are three important commands for navigating the file system via the shell:
cd path: Change the current directory to the one specified bypathor go to your home directory ifpathis omittedls path: List the contents of the directory at thatpathor the current directory ifpathis omittedpwd: Print to the screen your current working directory.
Here is a sample of using these commands to explore the file system:
aviv@zee:~$ pwd /home/scs/aviv aviv@zee:~$ ls aviv-local@ class/ Downloads/ local/ Public/ test.c VBox-Map/ #VM-notes.txt# #.bashrc# Desktop/ ic221/ Music/ Templates/ test.c~ Videos/ VM-notes.txt bin/ Documents/ id_rsa.pub Pictures/ test* tmp/ VirtualBox VMs/ VM-notes.txt~ aviv@zee:~$ cd tmp/ aviv@zee:~/tmp$ ls aviv@zee:~/tmp$ cd .. aviv@zee:~$ ls ~blenk ls: cannot open directory /home/scs/blenk: Permission denied aviv@zee:~$ ls class/ic221/ hw/ lab/ submit/ aviv@zee:~$ ls class/ic221/hw hw1.pdf
Note that the OS manages who can view what directories. When I tried to read someone elses home directory, I received a permission denied error.
aviv@zee: ~ $ ls ~m171110 ls: cannot access /home/mids/m171110: No such file or directory
This is an example of another important role of the operating system to provide security services on a shared systems. With multiple users and programs running on the same hardware base, it is the responsibility of the OS to ensure that users and programs do not access information they should not as well as not interfere with the execution of programs.
1.3.3 Understanding a Shell Prompt:
All shells have a command prompt (or just "prompt"), which indicates to the user to provide input. The prompt, itself, also provides some useful information about the shell, including things like the current working directory, your user name, and the host you are working on.
Here is an example prompt:
+ User Name
| +--Current Working Directory
| |
V V
aviv@zee:~$
^ ^ ^
Hostname-.__| | |_____.--- Where you enter commands
|
The prompt----+
You can see this command prompt change as you navigate the file system:
aviv@zee:~$ cd tmp/ aviv@zee:~/tmp$ ls aviv@zee:~/tmp$ cd .. aviv@zee:~$
While your shell may have a slightly different command prompt, all the same information is likely there. You can also change your command prompt as you want, and we will provide examples of doing that in later lessons.
1.4 File System Command Line Tools
Throughout this class, we will make use of a lot of standard Unix command line tools. These tools are common to nearly all Unix platforms. Above, we introduced three tools for navigating the file system, now we will explore some more, as well as their options.
1.4.1 Disecting a Command Line Argument
Some terminology regarding command line tools in the shell
+- Sell prompt, not included in the command
|
v
aviv@zee:~$ command arg1 arg2 arg3 ...
^ ^ ^ ^
| |_____|____|_____,-- The command arguments,
|
+-- The command, such as mv or cd
Most commands do not require arguments, but they are ways to provide a different set of information.
1.4.2 ls and it's arguments
First consider ls, which has a number of different options to display different listings of a directory.
ls path: list contents of directory atpathls -l path: long list the contents of the directory atpath, which includes permission, ownership, last edited, and file size.ls -a path: list all contents of directory atpathincluding hidden files that start with a ".", such as.bashrcls -al path: long list all contents of directory atpathincluding hidden files
aviv@zee:~$ ls aviv-local@ class/ Downloads/ local/ Public/ test.c VBox-Map/ #VM-notes.txt# #.bashrc# Desktop/ ic221/ Music/ Templates/ test.c~ Videos/ VM-notes.txt bin/ Documents/ id_rsa.pub Pictures/ test* tmp/ VirtualBox VMs/ VM-notes.txt~ aviv@zee:~$ ls -a ./ bin/ .dmrc .gnome2/ .launchpadlib/ .pulse-cookie Videos/ ../ .cache/ Documents/ .gnome2_private/ .lesshst .ssh/ VirtualBox VMs/ aviv-local@ class/ Downloads/ .gtk-bookmarks local/ .tcshrc #VM-notes.txt# .bash_history .compiz/ .emacs .gvfs/ .local/ Templates/ VM-notes.txt .bash_profile .compiz-1/ .emacs~ .hplip/ .mozilla/ test* VM-notes.txt~ .bash_profile~ .config/ .emacs.d/ ic221/ Music/ test.c .vmware/ .bashrc .cshrc .fontconfig/ .ICEauthority Pictures/ test.c~ .Xauthority .bashrc~ .dbus/ .gconf/ id_rsa.pub Public/ tmp/ .xsession-errors #.bashrc# Desktop/ .gksu.lock .inputrc .pulse/ VBox-Map/ .xsession-errors.old
Notice all the files that start with . that are now visible with the -a option, as well as . and ..
When using ls -l you get a lot of extra information.
aviv@saddleback: ~ $ ls -l total 484 -rw-r----- 1 aviv scs 85816 May 27 2014 302.packages.dat -rw-r----- 1 aviv scs 14284 May 27 2014 302.packages.installed.dat -rw------- 1 aviv scs 13524 Dec 23 2013 accesslog.gz -rw------- 1 aviv scs 115373 Dec 23 2013 authlog.dat lrwxrwxrwx 1 aviv scs 17 Nov 5 2013 aviv-local -> /local/aviv-local drwxr-xr-x 2 aviv scs 4096 Dec 17 15:07 bin drwxr-x--x 4 aviv scs 4096 Aug 18 11:10 class drwx--x--x 2 aviv scs 4096 Dec 22 2013 Desktop drwx--x--x 2 aviv scs 4096 Oct 17 2013 Documents drwx--x--x 2 aviv scs 4096 Dec 22 2013 Downloads drwxr-x--- 3 aviv scs 4096 Oct 23 11:42 etector drwxr-xr-x 2 aviv scs 4096 Apr 22 2014 final-practicum drwx------ 3 aviv scs 4096 Dec 23 2013 git drwxr-xr-x 4 aviv scs 4096 Feb 12 2014 GNUstep -rw-r----- 1 aviv scs 396 Mar 23 2014 guineapig.id_rsa.pub -rw-r----- 1 aviv scs 366 Feb 26 2014 helloworld.c -rw-r----- 1 aviv scs 358 Feb 26 2014 helloworld.c~ -rw-r----- 1 aviv scs 32109 Apr 10 2014 lab.html drwx--x--x 4 aviv scs 4096 Dec 24 2013 local drwxr-x--- 3 aviv scs 4096 Apr 23 2014 Mail -rw------- 1 aviv scs 185 Jan 16 2014 most_recent_sub.sh drwx--x--x 2 aviv scs 4096 Oct 17 2013 Music drwxr-x--- 4 aviv scs 4096 Dec 17 15:06 old-ic221 -rw-r----- 1 aviv scs 0 Mar 19 2014 output.txt drwx--x--x 2 aviv scs 4096 Dec 22 2013 Pictures drwx--x--x 2 aviv scs 4096 Jan 7 2014 Public drwxr-xr-x 4 aviv scs 4096 Nov 14 11:05 public_html -rw-r----- 1 aviv scs 85938 May 27 2014 saddleback.packages.dat -rw-r----- 1 aviv scs 14213 May 27 2014 saddleback.packages.installed.dat drwxr-x--- 4 aviv scs 4096 Aug 18 10:51 si221 drwxr-x--- 5 aviv scs 4096 May 27 2014 svn -rw-r----- 1 aviv scs 4530 Dec 23 2013 syslog.dat drwx--x--x 2 aviv scs 4096 Oct 17 2013 Templates drwx--x--x 4 aviv scs 4096 Dec 29 10:38 tmp drwx--x--x 2 aviv scs 4096 Nov 11 2013 VBox-Map drwx--x--x 2 aviv scs 4096 Oct 17 2013 Videos drwx--x--x 2 aviv scs 4096 Nov 5 2013 VirtualBox VMs -rw-r--r-- 1 aviv scs 465 Nov 8 2013 VM-notes.txt lrwxrwxrwx 1 aviv scs 12 Jan 9 2014 web -> public_html/
We can interpret this information as:
.- Directory?
| .-------Permissions .- Directory Name
| ___|___ .----- Owner |
v/ \ V ,---- Group V
drwxr-x--x 4 aviv scs 4096 Dec 17 15:14 ic221
-rw------- 1 aviv scs 400 Dec 19 2013 .ssh/id_rsa.pub
^ \__________/ ^
File Size -------------' | '- File Name
in bytes |
|
Last Modified --------------'
On your own: Try using the following variants of the ls command:
ls -hls -kls *
You may also notice some other kinds of files, for example:
lrwxrwxrwx 1 aviv scs 12 Jan 9 2014 web -> public_html/
This is a symbolic link, notice the l in the prefix. We'll discuss
these later in the semester. For now, understand this a lot like a
shortcut, web is a shortcut for the directory pubic_html/.
1.4.3 File System Manipulation Commands
So far, we've looked at commands for navigating the file system, now we are going to look at commands that can manipulate the file system. The most basic actions for file system manipulation and their corresponding commands are:
cp from to: Copy a file/directory from pathfromto pathtomv from to: Move a file/directory from pathfromto pathto, also used to change the name of a file/directoryrm path: Remove a file frompathmkdir path: Make a directory atpathtouch path: Create an empty file atpath
Let's look at a demo of doing this
aviv@zee:~/class/ic221/demo$ mkdir NewDir <--- Create a new directory aviv@zee:~/class/ic221/demo$ ls <--- Show it was created NewDir/ aviv@zee:~/class/ic221/demo$ cd NewDir/ <--- Change into that directory aviv@zee:~/class/ic221/demo/NewDir$ ls <--- List contents, it's empty aviv@zee:~/class/ic221/demo/NewDir$ touch foo.txt <--- Create an empty file foo.txt aviv@zee:~/class/ic221/demo/NewDir$ cp foo.txt baz.txt <--- Copy foo.txt to baz.txt aviv@zee:~/class/ic221/demo/NewDir$ ls <--- List contents of directry with foo.txt and baz.txt baz.txt foo.txt aviv@zee:~/class/ic221/demo/NewDir$ mv baz.txt .. <--- Move baz.txt to parent directory aviv@zee:~/class/ic221/demo/NewDir$ ls <--- List contents of directory, no baz.txt foo.txt aviv@zee:~/class/ic221/demo/NewDir$ cd .. <--- Change to partent directory aviv@zee:~/class/ic221/demo$ ls <--- List contents, show baz.txt and NewDir baz.txt NewDir/ aviv@zee:~/class/ic221/demo$ rm baz.txt <--- Remove baz.txt rm: remove regular empty file `baz.txt'? y <--- confirm it's removal aviv@zee:~/class/ic221/demo$ rm NewDir/foo.txt <--- Remove foo.txt by using a path to it rm: remove regular empty file `NewDir/foo.txt'? y <--- confirm it's removal aviv@zee:~/class/ic221/demo$ rm NewDir/ <--- Remove the direcotry rm: cannot remove `NewDir/': Is a directory <--- FAIL!
1.4.4 Handling Directories and Recursive (-r) Option
Note that rm cannot directly remove a directory, instead you
have to use a special form of remove, the rmdir command.
aviv@zee:~/class/ic221/demo$ rmdir NewDir/ aviv@zee:~/class/ic221/demo$ ls aviv@zee:~/class/ic221/demo$
However, you cannot rmdir if there is contents in the directory
aviv@zee:~/class/ic221/demo$ mkdir NewDir aviv@zee:~/class/ic221/demo$ touch NewDir/foo.txt aviv@zee:~/class/ic221/demo$ rmdir NewDir/ rmdir: failed to remove `NewDir/': Directory not empty
There is an option to remove, -r, which stands for recursive, that
will recursively remove a directory and its contents.
aviv@zee:~/class/ic221/demo$ rm -r NewDir/ rm: descend into directory `NewDir/'? y rm: remove regular empty file `NewDir/foo.txt'? y rm: remove directory `NewDir'? y aviv@zee:~/class/ic221/demo$ ls aviv@zee:~/class/ic221/demo$
Similar issues occur when you are trying to copy a directory with
cp, you need to specify the recursive option "-r". As you can
see from the demo below, this also copies the entire contents of the directory.
aviv@zee:~/class/ic221/demo$ mkdir NewDir aviv@zee:~/class/ic221/demo$ touch NewDir/foo.txt aviv@zee:~/class/ic221/demo$ ls NewDir/ aviv@zee:~/class/ic221/demo$ cp NewDir/ CopyDir cp: omitting directory `NewDir/' aviv@zee:~/class/ic221/demo$ ls NewDir/ aviv@zee:~/class/ic221/demo$ cp -r NewDir/ CopyDir aviv@zee:~/class/ic221/demo$ ls CopyDir/ NewDir/ aviv@zee:~/class/ic221/demo$ ls CopyDir/ foo.txt
The move command does not require a recursive option when interacting with directories.
aviv@zee:~/class/ic221/demo$ ls CopyDir/ NewDir/ aviv@zee:~/class/ic221/demo$ mv NewDir/ CopyDir/ aviv@zee:~/class/ic221/demo$ ls CopyDir/ aviv@zee:~/class/ic221/demo$ ls CopyDir/ foo.txt NewDir/ aviv@zee:~/class/ic221/demo$ ls CopyDir/NewDir/ foo.txt
1.5 Where Commands "Live"
Now you are more familiar with navigating and manipulating the Linux file system, let's return to the basic structure of the Linux root file system.
When you type the command ls or rm these commands are really
program binaries that have to exist somewhere in the file
system. Since these are binaries, by convention they exist in a
directory that ends in bin. The way the shell finds these commands is
by searching through a sequence of bin folders until it finds it.
The search path for binaries is called the $PATH or just path. You
can display your current path using the echo command, which just
prints to the screen.
aviv@zee:~$ echo $PATH /home/scs/aviv/bin:/opt/local/bin:/opt/local/sbin:usr/lib/jvm/java-6-sun/bin:/home/scs/aviv/bin:
So when you type a command like ls, the shell looks in each of the
folders for a program named ls to run. It happens that ls
exists in the base /bin folder, which means you can run it in
shorthand and using its full path:
aviv@zee:~$ ls aviv-local@ class/ Downloads/ local/ Public/ test.c VBox-Map/ #VM-notes.txt# #.bashrc# Desktop/ ic221/ Music/ Templates/ test.c~ Videos/ VM-notes.txt bin/ Documents/ id_rsa.pub Pictures/ test* tmp/ VirtualBox VMs/ VM-notes.txt~ aviv@zee:~$ /bin/ls aviv-local class Downloads local Public test.c VBox-Map #VM-notes.txt# #.bashrc# Desktop ic221 Music Templates test.c~ Videos VM-notes.txt bin Documents id_rsa.pub Pictures test tmp VirtualBox VMs VM-notes.txt~
1.5.1 The which command
Unix provides a command line utility for finding where a command
lives, the which command.
aviv@zee:~$ which ls /bin/ls aviv@zee:~$ which rm /bin/rm aviv@zee:~$ which which /usr/bin/which
Most basic commands that are part of the Base system are in /bin
but user system resources (usr) command line tools exist in
/usr/bin, like which itself. What other commands have we looked
at is in /usr/bin?