Linux Basics

Linux Basics#

This notebook serves as starting point for learning linux and the bash terminal. I highly recommend, that you run it interactively via the mybinder.org link in the top. This will allow you to execute the commands yourself.

Usually developers, system administrators, devop engineers, etc. interact with the the linux operating system via a textual interface, called the shell. The most common type of shell is called bash. For now, the differences between these is not so important, but during troubleshooting it might help to know which shell you are using.

Linux comes with lots of preinstalled programs which make working with the shell very easy. We will look at some of them and how to use them.

This notebook is for learning purpose only. You will never interact with the shell using such a notebook. The %%bash as well as the exclamation mark (!) in the lines is only for the notebook. The %%bash transforms the whole cell into a bash terminal while the ! runs only a single command via the shell.

File System#

Linux is a directory/file-based operating system. In Linux, everything is treated as a file, including hardware devices, processes, and system configurations. The file system is organized in a hierarchical directory structure, with the root directory / at the top. All files and directories are accessed relative to this root directory.

The home directory is a personal directory assigned to each user on the system. It serves as the default working directory when a user logs in and is where the user’s personal files, configurations, and directories are stored. The ~ is a shorthand for the home directory. It simplifies navigation.

Typically, when you open up a terminal, your session will start in the ~ folder. This is not always the case, as for the notebook here for example. The notebook and the shell open up in the folder from where they are launched. Let us look at that first.

!pwd     # print the current working directory
!ls      # list the contents of the current working directory

/home/runner/work/learning-notes/learning-notes/learning-notes
_config.yml	   dqn.ipynb		 kubernetes.md	      pca.ipynb
_toc.yml	   formular.ipynb	 links.md	      perceptron.ipynb
abbreviations.md   images		 linux.ipynb	      rl.ipynb
assessment.md	   intro.md		 mlp.ipynb	      rnn.ipynb
bash.md		   iris.ipynb		 obj-detection.ipynb  softmax.ipynb
cnn.ipynb	   k-means.ipynb	 optimizers.ipynb     torch.ipynb
convolution.ipynb  kalman-filters.ipynb  p-np.ipynb	      virtualization.md

Next we want to see how we can navigate the filesystem. Let us start with the root and see what lies beneath it. We can chain commands via the logical AND operator &&. Only if the first command succeeds, the second command will be run.

!cd / && ls  # change to the root directory and list its contents

bin		   etc		      lib32	  opt	sbin.usr-is-merged  usr
bin.usr-is-merged  home		      lib64	  proc	snap		    var
boot		   imagegeneration    lost+found  root	srv
data		   lib		      media	  run	sys
dev		   lib.usr-is-merged  mnt	  sbin	tmp

While the root directory is the starting point, you will seldomly operate inside it directly. You will however use it for navigation. The Github build pipeline runs this notebook inside an Ubuntu linux, such the output here will reflect a typical ubuntu installation. Ubuntu follows the Filesystem Hierarchy Standard. It is not necessary to remember them all, but it is good to have an overview of the most important directories.

  • /dev: Contains device files that represent hardware devices, such as sda (hard drives), tty (terminals), and null. etc

  • /tmp: A temporary directory for storing files that are deleted after a reboot or after a certain period.

  • /var: Contains variable data, such as logs (log), mail (mail), and temporary files for running services.

  • /home: Contains the home directories of users. For example, /home/alice is the home directory for the user alice.

  • /usr/bin: Non-essential user binaries.

  • /usr/lib: Libraries for user applications.

  • /mnt: A temporary mount point for attaching external storage devices like USB drives or network shares.

!whoami       # shows my username
!cd ~ && pwd  # change to my personal directory and show the current working directory

runner
/home/runner

As you can see, our home directory lies beneath /home/. This is expected.

Manipulating files#

As said before, everything is file based in linux. A common task is to edit configuration files or to read logs etc.

!touch mytext.txt             # create a new file called mytext.txt in the root directory
!echo "Hello" > mytext.txt    # overwrite the file with the word "Hello"
!echo "World!" >> mytext.txt  # append "World!" to the file
!cat mytext.txt               # display the contents of the file

Hello
World!

We have seen a lot. touch simply creates a a file. We give it the txt suffix to highlight that it is a plain text file. Although the suffix does not mean a lot. It is just an indication. Next, we used the echo command to output text. A plain echo "Hello" would just print Hello to the standard outoutput stdout, which is typically the terminal. The > operator redirects the output to a file instead of displaying it to the terminal. The >> appends to a file instead of overwriting. If a program prints a lot of debug information for example to stdout, you can use >/>> operator to aggregate the logs and write them into a file.

Man Pages#

Programs come very often with a manual. The man xyz opens the manual. There you can find additional command line parameters and explanations. For the cat command you will also find that there is a –help flag which we can use.

!man cat

CAT(1)                           User Commands                          CAT(1)

NAME
       cat - concatenate files and print on the standard output

SYNOPSIS
       cat [OPTION]... [FILE]...

DESCRIPTION
       Concatenate FILE(s) to standard output.

       With no FILE, or when FILE is -, read standard input.

       -A, --show-all
              equivalent to -vET

       -b, --number-nonblank
              number nonempty output lines, overrides -n

       -e     equivalent to -vE

       -E, --show-ends
              display $ at end of each line

       -n, --number
              number all output lines

       -s, --squeeze-blank
              suppress repeated empty output lines

       -t     equivalent to -vT

       -T, --show-tabs
              display TAB characters as ^I

       -u     (ignored)

       -v, --show-nonprinting
              use ^ and M- notation, except for LFD and TAB

       --help display this help and exit

       --version
              output version information and exit

EXAMPLES
       cat f - g
              Output f's contents, then standard input, then g's contents.

       cat    Copy standard input to standard output.

AUTHOR
       Written by Torbjorn Granlund and Richard M. Stallman.

REPORTING BUGS
       GNU coreutils online help: <https://www.gnu.org/software/coreutils/>
       Report any translation bugs to <https://translationproject.org/team/>

COPYRIGHT
       Copyright  ©  2023  Free Software Foundation, Inc.  License GPLv3+: GNU
       GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
       This is free software: you are free  to  change  and  redistribute  it.
       There is NO WARRANTY, to the extent permitted by law.

SEE ALSO
       tac(1)

       Full documentation <https://www.gnu.org/software/coreutils/cat>
       or available locally via: info '(coreutils) cat invocation'

GNU coreutils 9.4                 April 2024                            CAT(1)

If you read the man page, you will see

cat - concatenate files and print on the standard output

Again, the standard output. This means we can also redirect the output of the file to another file using >.

!cat mytext.txt > mytext2.txt
!cat -n mytext2.txt

     1	Hello
     2	World!

Pipes#

In Bash, pipes (|) are used to pass the output of one command as the input to another command. This allows you to chain commands together to perform complex operations efficiently.

command1 | command2

  1. command1 is executed, and its output is sent to the standard output (stdout).

  2. Instead of displaying the output, the pipe (|) redirects it as input to command2.

  3. command2 processes the input and produces its own output.

!ls | grep "mytext" # list all files in the current directory and filter for those containing "mytext"
!ps aux | grep "python"  # list all processes and filter for those containing "python"

mytext.txt
mytext2.txt
root         824  0.0  0.1  32560 21044 ?        Ss   18:32   0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
root         854  0.1  0.2  44032 35672 ?        Ss   18:32   0:00 /usr/bin/python3 -u /usr/sbin/waagent -daemon
root        1527  0.2  0.2 451976 35032 ?        Sl   18:33   0:00 /usr/bin/python3 -u bin/WALinuxAgent-2.13.1.1-py3.9.egg -run-exthandlers
runner      2022  1.3  0.8 498552 139076 ?       Sl   18:36   0:02 /opt/hostedtoolcache/Python/3.11.12/x64/bin/python /opt/hostedtoolcache/Python/3.11.12/x64/bin/jupyter-book build learning-notes --path-output book
runner      2297 10.7  0.4 768620 67740 ?        Ssl  18:38   0:00 /opt/hostedtoolcache/Python/3.11.12/x64/bin/python -m ipykernel_launcher -f /tmp/tmpps2zgreh.json --HistoryManager.hist_file=:memory:
runner      2351  0.0  0.0   2808  1748 pts/0    Ss+  18:38   0:00 /usr/bin/sh -c ps aux | grep "python"  # list all processes and filter for those containing "python"
runner      2353  0.0  0.0   7084  2044 pts/0    S+   18:38   0:00 grep python

File Permissions#

In Linux, every file and directory has a set of permissions that control who can read, write, or execute them. These permissions are assigned to three categories of users:

  1. Owner: The user who owns the file.

  2. Group: A group of users who share access to the file.

  3. Others: Everyone else on the system.

Each file or directory can have the following permissions:

Read (r):

  • For files: Allows viewing the contents of the file.

  • For directories: Allows listing the contents of the directory.

Write (w):

  • For files: Allows modifying or deleting the file.

  • For directories: Allows creating, renaming, or deleting files within the directory.

Execute (x):

  • For files: Allows running the file as a program or script.

  • For directories: Allows accessing the directory (e.g., using cd).

!ls -l

total 340
-rw-r--r-- 1 runner docker   834 May 13 18:35 _config.yml
-rw-r--r-- 1 runner docker   884 May 13 18:35 _toc.yml
-rw-r--r-- 1 runner docker 24278 May 13 18:35 abbreviations.md
-rw-r--r-- 1 runner docker  2078 May 13 18:35 assessment.md
-rw-r--r-- 1 runner docker  1696 May 13 18:35 bash.md
-rw-r--r-- 1 runner docker 18400 May 13 18:35 cnn.ipynb
-rw-r--r-- 1 runner docker 18814 May 13 18:35 convolution.ipynb
-rw-r--r-- 1 runner docker 13477 May 13 18:35 dqn.ipynb
-rw-r--r-- 1 runner docker  4370 May 13 18:35 formular.ipynb
drwxr-xr-x 2 runner docker  4096 May 13 18:35 images
-rw-r--r-- 1 runner docker  1206 May 13 18:35 intro.md
-rwxr-xr-x 1 runner docker  4930 May 13 18:35 iris.ipynb
-rw-r--r-- 1 runner docker  8346 May 13 18:35 k-means.ipynb
-rw-r--r-- 1 runner docker 12301 May 13 18:35 kalman-filters.ipynb
-rw-r--r-- 1 runner docker  6069 May 13 18:35 kubernetes.md
-rw-r--r-- 1 runner docker   505 May 13 18:35 links.md
-rw-r--r-- 1 runner docker 13369 May 13 18:35 linux.ipynb
-rw-r--r-- 1 runner docker 16728 May 13 18:35 mlp.ipynb
-rw-r--r-- 1 runner docker    13 May 13 18:38 mytext.txt
-rw-r--r-- 1 runner docker    13 May 13 18:38 mytext2.txt
-rw-r--r-- 1 runner docker 10068 May 13 18:35 obj-detection.ipynb
-rw-r--r-- 1 runner docker 13307 May 13 18:35 optimizers.ipynb
-rw-r--r-- 1 runner docker 11359 May 13 18:35 p-np.ipynb
-rw-r--r-- 1 runner docker 14891 May 13 18:35 pca.ipynb
-rw-r--r-- 1 runner docker 16219 May 13 18:35 perceptron.ipynb
-rw-r--r-- 1 runner docker 17393 May 13 18:35 rl.ipynb
-rw-r--r-- 1 runner docker 12666 May 13 18:35 rnn.ipynb
-rw-r--r-- 1 runner docker  4262 May 13 18:35 softmax.ipynb
-rw-r--r-- 1 runner docker 12570 May 13 18:35 torch.ipynb
-rw-r--r-- 1 runner docker  3957 May 13 18:35 virtualization.md

An example output looks like this:

-rwxr-xr--  1 user group 1234 May 12 12:00 file.txt

  • The first character (-) indicates the type of file (- for regular file, d for directory).

  • The next nine characters are the permissions:

    • rwx (Owner): Read, Write, Execute.

    • r-x (Group): Read, Execute.

    • r-- (Others): Read only.

Using chmod, we can modify the permissions. Lets change the permissions for mytext.txt

Add, remove, or set permissions using u (owner), g (group), o (others), or a (all). Use + (add), - (remove), or = (set).

!chmod a=w mytext.txt   # change permissions to allow all users to write
!ls -l mytext.txt
!chmod a=r mytext.txt   # change permissions to allow all users to read
!ls -l mytext.txt
!chmod a+x mytext.txt   # change permissions to allow all users to execute
!ls -l mytext.txt
!chmod a-x mytext.txt   # change permissions to remove execute permissions for all users
!ls -l mytext.txt
!chmod u+x mytext.txt   # change permissions to allow the user (owner) to execute
!ls -l mytext.txt
!chmod g+x mytext.txt   # change permissions to allow the group to execute
!ls -l mytext.txt
!chmod o+x mytext.txt   # change permissions to allow others to execute
!ls -l mytext.txt

--w--w--w- 1 runner docker 13 May 13 18:38 mytext.txt
-r--r--r-- 1 runner docker 13 May 13 18:38 mytext.txt
-r-xr-xr-x 1 runner docker 13 May 13 18:38 mytext.txt
-r--r--r-- 1 runner docker 13 May 13 18:38 mytext.txt
-r-xr--r-- 1 runner docker 13 May 13 18:38 mytext.txt
-r-xr-xr-- 1 runner docker 13 May 13 18:38 mytext.txt
-r-xr-xr-x 1 runner docker 13 May 13 18:38 mytext.txt

As you can see, there are multiple ways to set permissions for the files. Setting correct permissions is an important tasks, since you do not want your configuration to be manipulated by everyone. Usually, they are heavily restricted, such that only an administrator can change them.

While the symbolic way is one option to change the permission, there is another more effective way, the numeric mode.

  • 4 = Read (r)

  • 2 = Write (w)

  • 1 = Execute (x)

Each user/group/all permission is assigned a combiniation of these three numbers. This is usually a lot simpler, sice you can assign the correct permissions in one step:

!chmod 777 mytext.txt  # change the permissions of mytext.txt to read, write, and execute
!ls -l mytext.txt  # list the details of mytext.txt to see the new permissions
!chmod 644 mytext.txt  # change the permissions of mytext.txt back to read and write for the owner, and read for others
!ls -l mytext.txt  # list the details of mytext.txt to see the new permissions

-rwxrwxrwx 1 runner docker 13 May 13 18:38 mytext.txt
-rw-r--r-- 1 runner docker 13 May 13 18:38 mytext.txt
Contents