Familiarizing with bash

This blog contains some notes for using bash command. I’m writing them here in an “aid-sheet” style. This could also be a concise, elementary tutorial into bash. I’ll group them into following sections.
Survival

  • File system navigation (cd, ls, cat, du, df)
  • Playing around with variables

Remote working

  • Connect to remote computer (ssh, sftp, wget, curl)
  • Compress / uncompress files (dpkg, tar, zip)
  • Run programs remotely (chmod, nohup, &, &&)

Processing data

  • For loop, sed, find, grep, piping

Survival

0. RTFM

Goal Command
Read the manual of command command_name man command_name
Find the command for a specific purpose Search for it online

For any command mentioned below, you can refer to a comprehensive documentation at man <command_name>. This blog can only provide a very short description, for easiness of memorization. The man page is the textbook, while this blog is at most an aid-sheet.

1. Navigating your filesystem

Goal Command
To navigate to your home folder cd ~/
To navigate to the previous directory cd -
Create a folder with name myname mkdir myname
View all items in current folder ls
View, and with information including permissions, file sizes ls -l
View, while making the file sizes human readable ls -lh
View the size of the current folder du
View, while making the file sizes human readable du -h
Check file system space usage df
View the content of the file myfile.txt cat myfile.txt
View the contents of files myfile1.txtmyfile3.txt cat myfile[1-3].txt
View the contents of all files under current directory matching regular expression myfile*.txt cat myfile*.txt
Write them to mylog.txt while overwriting this txt file cat myfile*.txt > mylog.txt
Append them to the end of ~/.bashrc cat myfile*.txt >> mylog.txt

2. Playing around with variables

Goal Command
To view the value of variable JAVA_HOME echo $JAVA_HOME
Evaluate the value of bash_command, and print it out echo $bash_command
To change the value of variable JAVA_HOME to /usr/local/java JAVA_HOME=/usr/local/java
To change it at startup (i.e., append to ~/.bashrc or ~/.profile) echo "JAVA_HOME=/usr/local/java" >> ~/.bashrc
To change it via a script setup_env.sh source setup_env.sh (Shouldn’t use sh since that creates another shell, which has its own set of environment variables.)

Remote working

3. Working with a remote computer: ssh, sftp, wget, curl

Goal Command
To establish an ssh connection ssh <username>@<address> -p <port_number>
To establish an sftp connection to your directory sftp -P <port_number> <username>@<address>:<directory>
(during an sftp connection) Download txt files with file name starting with test_ get test_*.txt
To download an object with wget from url https://somewhere/something.tar.gz wget https://somewhere/something.tar.gz
To submit GET request with curl curl https://somwehre
To submit a POST request with curl curl --data "name1=value1&name2=value2" https://somewhere

4. Zipping with tar, gzip, zip

Goal Command
To extract (-x) the .tar.gz file here, verbose (-v) tar -xzvf something.tar.gz
To create (-c) .tar.gz file tar -czvf something.tar.gz myfile1.txt
To extract the tar file tar -xvf something.tar.gz
To create a file into tar file tar -cvf something.tar myfile1.txt
To unzip the .zip file unzip something.zip
To zip files matching regexp myfile[1-3].txt into myzip zip myzip myfile[1-3].txt

5. Run a program

Goal Command
Make the file tmp.sh runnable chmod +x tmp.sh
Run command in the current process ./tmp.sh
Run it as a child process of current process (i.e., will be killed when you log out) ./tmp.sh &
Run it, but without it being killed, and write stdout to your_log.out nohup ./tmp.sh > your_log.out &
Run multiple commands using AND operator (i.e., &&) command1 && command2
Input prepared stdin data tmp.sh < prepared_stdin.in

Processing data

For loop

Goal Command
Loop over an array and print the values for i in {1,3,5}; do echo $i; done
Loop over a range and print the values for i in {1..5}; do echo $i; done

grep, pipe, sed, find

Goal Command
Display all .txt files under current directory ls | grep *.txt
Display files, but applying multiple grep filters ls | grep | grep
Count the number of txt files ls | grep *.txt | wc -l
Display only (-o) the numerical values from the variable output echo $output | grep -oE [0-9]
Replace, in-place (-i) <regex> to “Content” of the file readme.md sed -i "s/<regex>/Content/" readme.md
Find a file under current directory by filename find . -name <regex>
Find content in directory dirname recursively (-R), with line number (-n), case-insensitive (-i) containing CrossEntropy grep CrossEntropy dirname -iRn