Linux

Linux principles:

• everything in Linux is a file
• small a single purpose programs
• chain programs together to perform a complex purpose
• avoid user interface
• configuration in a text file

Terms:

Shell =>Kernel=> Hardware

Bash is a sort of Shell, other shells, fx: Fish, Zsh, Ksh

A Terminal, also called a shell, is a GUI to interact with the kernel.

Linux folder

/(forward slash): Root 

/home/username: each user has a personal folder

/bin: command binaries. 

/etc: local system configuration files

/lib: shared libs shared for system boot

/opt: the third-party tools

/var: variable files, fx: log files

Linux commands

ls -a: list all files including hidden ones.

ls -l: list files in a long format.

ls -l *.txt

 

cd: change the current directory

.. indicates the parent folder

mkdir: make a new folder

mkdir folder1 folder2: creating several folders

mkdir -p folder/nested-folder: creating m-nested folders

rmdir: remove a folder

rm: remove one file or multiple files

rm blue red yellow

rm -v blue red yellow 

-v: verbose.  -r: recursive -i: interactive -f: force

rm -rf folder-name

touch: if the file is not existed, create one, else update to a new timestamp. 

touch new-file-name

open: open a file or directory just like double-clicked file icon. 

open. open the current folder in a graphic presentation

open file-name: open the file in OS default text editor

> : redirect output console

date > newFile.text: output current date to designated file, and overwrite the file entirely.

>>: redirect out the console and appending

date>> newFile.text: out current date to designated file, and appending. 

cp: copy file or folder

less: a UI showing the entire file, scrolling down and up.

less file-name

WC: word count

wc -l file-name: shows line number in a file

wc -w file-name: shows how many words are in a file

piping |: taking one command output as input of another command. 

Env var

expending dollar sign or tilde

dollar sign path

echo * all path names

echo *.txt

? matching each single character

cry0l1t3@htb[/htb]$ ls -l /etc/passwd

- rwx rw- r--   1 root root 1641 May  4 23:42 /etc/passwd
- --- --- ---   |  |    |    |   |__________|
|  |   |   |    |  |    |    |        |_ Date
|  |   |   |    |  |    |    |__________ File Size
|  |   |   |    |  |    |_______________ Group
|  |   |   |    |  |____________________ User
|  |   |   |    |_______________________ Number of hard links
|  |   |   |_ Permission of others (read)
|  |   |_____ Permissions of the group (read, write)
|  |_________ Permissions of the owner (read, write, execute)
|____________ File type (- = File, d = Directory, l = Link, ... )

chmod 777 file-name : change file read-write-executable

find . -name fileName: find the current folder by name 

which: shows the path to a file

which file-name: locate a file

ps: shows all processes

ps -ef: extended format 

ps -ef | grep bash

kill PID: kill by its process ID

top: show statistics of system usage

wget URI: download resources

su: switch user to log in as someone else

exit: log out a user account

sudo: super user do, run a command as a root user

sudo apt update:  update the APT package index in line with the repository.

sudo apt upgrade: update the installed packages to the latest version.

sudo apt upgrade specific-package: update an installed package to the newest version.

sudo apt install package-name: install package 

source file-name [argument]: shell built-in command that can execute a file containing a list of shell commands. 

export: built-in shell command to mark env variable to export the child processes(change visibilities). 

sh xxx.sh: execute a script file

Vi/Vim Editor

vi target_file_name

entry mode| edit mode | command mode

press esc, and then press key i, then seeing bottom showing 'Insert'; now in an edit mode

press esc, then quit from insert model, then in an entry mode

press esc, then press :, then enter command mode.

press esc, bottom right input commands

:w save current file

:q quit from file without saving

:wq  write and quit from current file

:q! force to quit without saving

:set nu showing line number

:set nonu turning off line number

Network

ping domain_name or ip_address

Check Remote Ports are Reachable Using ‘nc’ Command

Using netcat, you can check if a single or multiple or a range of open ports as follows. The command below will help us see if the port 22 is open on the host 192.168.56.10:

nc -zv 192.168.1.15 22

-z – sets nc to simply scan for listening daemons, without actually sending any data to them.
-v – enables verbose mode.

The next command will check if ports 80, 22 and 21 are open on the remote host 192.168.5.10 (we can use the hostname as well):
nc -zv 192.168.56.10 80 22 21

and it is also possible to check a range of ports.

nc -zv 192.168.56.10 20-80

how to install JDK from Linux 

Concurrent Programming Review

Process and Thread

The process is self-contained. It doesn't share the resource with others. A thread is a minimal execution unit, living inside a process. One process may consist of many threads. Threads may share a common resource with each other.

Volatile: Solving the visibility issue between threads.

Each Thread has its own local cache, so the variable defined in a thread won't be visible by another thread. When using Volatile keyword to modify a variable, the OS will push the variable into the shared cache(among the threads), thus it becomes visible to all other threads.

Volatile provides visibilities between isolated threads, however it doesn't provide an atomic protection from multiple step operation(read-modification-write).

Concurrent and Parallel computing 

Concurrent computing task can be decomposed into sub-tasks, which are order-independent.  These sub-tasks are carried out at the same core but different threads. Form an external view, it seems that these tasks are carried out simultaneously, but actually not.

Parallel computing needs parallel computing hardware first, i.e. having more than one core. The computing task is carried out in independent cores simultaneously.  

Mutex

Mutex means mutual exclusion is the way to solve the data race. At any time, only one thread is allowed to access the shared data. 

Data race: 

Simultaneously multiple threads access a common memory location.  The data stored there may be not consistent and being polluted. Ideally,  all steps should be carried out in a single uninterruptible operation, so as to prevent data from pollution. 

Lock: 

Two types of locks, i.e. extrinsic or intrinsic lock.

An extrinsic lock is explicitly declared from Lock interface and its implementations. A critical section is protected by an explicit lock and afterwards, the lock is unlocked.

Synchronization: 

Two types of synchronization, i.e. synchronization method or synchronization statements.

Every Java class or object has an intrinsic lock associated with it. The intrinsic lock must be achieved first, so as to access the synchronized method or statements.

Note: ➀watching out Synchronizing on the class or instance; ➁ don't synchronize on an immutable object, for it generates a new instance after its value being modified.

Atomic variables:

The most commonly used atomic variable classes in Java are AtomicInteger, AtomicLong, AtomicBoolean, and AtomicReference. These classes represent an int, long, boolean and object reference respectively which can be atomically updated.

For instance, AtomicInteger provides a method incrementAndGet(), it is not synchronized, but it offers
atomicity guarantees by using Low-level compare-and-set(CAS) instruction. 

Try Lock:

The tryLock method offers a non-blocking way to acquire the lock(explicit lock), so as to improve performance.

When a thread wants to acquire the lock, but not available, it may go away to carry out some other tasks, rather than being blocked and waiting there. This is the purpose to have 'try lock' method.

Read-write lock:

Instead of locking the resource to both read and write operations equally. A read-write lock is distinguished into a two-part lock, i.e. a read-alone lock and a write-alone lock. By this way, it may improve the performance. By this way, it allows multiple threads to read the shared resources simultaneously, for reading doesn't cause data race. It allows a single thread to write on the shared resource at a moment. This type of lock may be used when the number of reading threads is much bigger than the number of writing thread.

Note: read-write lock may use more resource under the hood.

Liveness

Deadlock:

Resource A and its Lock A; Resource B and its Lock B. Thread A hold Lock A, while Thread b hold Lock B.  Thread A acquires Lock B, while thread B acquires Lock A. 

How to prevent threads from deadlock?
lock priority: multiple threads acquire locks in a fixed order.
lock timeout: if cannot acquire, then quit.

Abandoned lock: 

when one thread acquired the lock, but it fails to unlock it due to exceptions. The lock is therefore abandoned. This may lead to the rest of the threads in stuck.

How to prevent a lock becoming abandoned?

using try-catch block to surround the critical section, and in the 'finally' block unlock the lock.

Starvation:

Starvation happens when a thread is perpetually denied resources due to competition with others. A thread may in starvation because it has a low priority. Or it simply because there are too many concurrent threads.

Livelock: 

Multiple threads are designed to respond to each other so as to solve the conflict.

To solve the livelock, making sure only one thread takes action. It may take use of random selection.

Explicit Lock vs Implicit Lock

Explicit locks, such as those provided by the java.util.concurrent.locks package (e.g., ReentrantLock, ReadWriteLock), offer several advantages over using intrinsic locks provided by synchronized blocks or methods. Some of the key benefits of using explicit locks are:

1. Flexibility: Explicit locks offer more flexibility in terms of lock acquisition and release. You can lock and unlock the lock object at different scopes and in separate blocks of code, which is not possible with synchronized blocks or methods.

2. Fairness: ReentrantLock allows you to specify fairness when creating the lock object. When a lock is fair, it grants access to the longest-waiting thread. Intrinsic locks provided by synchronized blocks or methods do not have a fairness option and may lead to thread starvation in some scenarios.

java
Lock lock = new ReentrantLock(true); // Create a fair lock

3. Try to acquire lock: Explicit locks provide the tryLock() method, which allows a thread to attempt to acquire the lock without waiting indefinitely. This can help avoid deadlocks and improve performance in some cases.

java
if (lock.tryLock()) {
    try {
        // Perform the operation
    } finally {
        lock.unlock();
    }
} else {
    // Handle the lock not being acquired
}

4. Interruptible lock acquisition: With explicit locks, you can use the lockInterruptibly() method, which allows a thread to be interrupted while waiting to acquire the lock. Intrinsic locks do not support this functionality.

java
try {
    lock.lockInterruptibly();
    try {
        // Perform the operation
    } finally {
        lock.unlock();
    }
} catch (InterruptedException e) {
    // Handle the interruption
}

5. Separate read and write locks: The ReadWriteLock interface provides separate locks for read and write operations, allowing multiple threads to read shared data simultaneously while still providing exclusive access for write operations. This can lead to better performance in scenarios with more reads than writes.

java
ReadWriteLock rwLock = new ReentrantReadWriteLock();
Lock readLock = rwLock.readLock();
Lock writeLock = rwLock.writeLock();

While explicit locks offer several advantages, they also require more careful handling and can be more error-prone due to the need for manual lock management. When choosing between explicit locks and intrinsic locks, consider the specific use case, the complexity of the code, and the required functionality. 

URL Special Signs

For example, a URL that includes a query string parameter for a search term might look like this:

https://www.example.com/search?term=best+restaurants

In this URL, the plus sign is used to represent the space between "best" and "restaurants" in the search term "best restaurants". When the web server receives this request, it would interpret the plus sign as a space and perform the search accordingly.

If you need to include a plus sign (+) as a literal character in a URL, you can use URL encoding to represent it as %2B.

URL encoding is a process in which special characters are replaced with their corresponding hexadecimal ASCII codes. This allows them to be transmitted over the internet without interfering with the meaning of the URL.

So if you want to include a plus sign in a URL, you can replace it with %2B. For example, the following URL includes a literal plus sign in the parameter value:

https://www.example.com/search?query=2+2=4%2B2

In this case, the plus sign between "4" and "2" is represented by %2B, which is the URL-encoded value of the plus sign character. When the web server receives this request, it would interpret the %2B code as a plus sign and perform the search accordingly.

Good Practice of Creating URI from String

It is generally recommended to use the java.net.URI class to create and manipulate URIs rather than constructing URIs directly from strings. This is because the URI class provides a higher level of abstraction and ensures that the resulting URI is valid and conforms to the URI syntax rules.The URI class provides constructors and methods to create and manipulate URIs, and it automatically handles any encoding and decoding that needs to be done. For example, if you want to create a URI from a string that contains special characters, you can use the URI(String) constructor as follows:

URI uri = new URI("http", "example.com", "/path with spaces", "query=string", "fragment");

This will create a URI with the following components:

http://example.com/path%20with%20spaces?query=string#fragment

Note that the path component, which contains spaces, is automatically encoded as %20.When creating a URI from user input, it is important to validate the input and ensure that it does not contain any invalid characters or sequences. This can be done using the java.net.URI.create(String) method, which will throw an IllegalArgumentException if the input string is not a valid URI.Overall, using the URI class provides a more robust and reliable way to create and manipulate URIs in Java.

MySql Database

1. Database In General

Relation database, Document database. 

Database Table Relationships

One to one:

mainly used for optimize a big table into two, one table contains often-visited columns, and another one contains rarely-visited columns. By this way, min. IO operations, i.e. reducing the amount data set that need to be loaded from persistences to memory.   

One to many: 

one entry ref. to many entries in another table

Many to many:

mutually ref. to many entries.

Self-reference:

entries ref. to anther entry located at the same table. 

Appendix 1. 

When we use SQL database, and when we select NoSQL database

SQL (Structured Query Language) databases and NoSQL databases are both types of databases that can be used to store and manage data. The choice between the two depends on several factors, including:

SQL databases are best suited for structured data, meaning data that can be organized into tables with defined relationships between the tables. This type of data is well suited for tasks such as financial reporting, where data is typically highly organized and predictable. SQL databases are also well suited for applications that require complex transactions, such as banking or e-commerce, where data consistency and integrity is critical.

NoSQL databases, on the other hand, are best suited for unstructured or semi-structured data, meaning data that doesn't fit neatly into a fixed table structure. NoSQL databases are often used for big data and real-time web applications, where data is generated at high volumes and speeds and needs to be processed and analyzed quickly. They are also well suited for cloud-based applications and distributed systems, where horizontal scalability is important.

In summary, the choice between SQL and NoSQL databases depends on the type of data you need to store and the demands of your application. If you have structured data and need to support complex transactions, an SQL database may be the best choice. If you have large amounts of unstructured data and need to handle real-time data processing, a NoSQL database may be a better fit.

 
 

Git commands

git-shell commands:

Git Local Config

git config --global user.email: showing current git user default email. 

git config --global user.name "John Doe": config git user name 

git config --list : check your local git config 

As committing
git init: creating a local git repository, i.e. .git in the current folder.
git add . : adding all files in a current folder for commit
git add filename.* : adding a specific file in the current folder for commit
git commit -m 'another commit' : commit added files to the local repository with a commit message.
git commit -a : commit all changes in the current dir.
git commit -am "message": commit all changes with a message in the current dir.
git remote add origin https://github.com/yichunzhao/Java8.git : link local repository to remote repository.
git push -u  dst  src ; git updating remote with a local source; -u: update

git remote rm origin: disconnect the link with the remote repository.

git pull: it is a shorthand for git fetch + git merge fetch_head

Info. 
git status: current branch status.
git remote -v  v: verbose displaying info. of remote repository

Branch 
git branch -r: remote branches
git branch: local branches

git branch -d|D branch-name: removing a local branch

git branch -d|D combined with -r branch-name: removing a remote branch

Push 
git push --set-upstream origin red: push local feature branch to the remote, when at the first time.

Stash 

git stash

git stash show

git stash show stash@{1}

git stash list : list all stash

git stash apply: applying the most recent stashed changes

Or

git stash pop : applying the most recently changes to your repo. 

Creating a new branch
git branch dev
git checkout master:  switching to another branch.

View history
git log :
git log --oneline
git reflog
git log --all --graph --decorate --online
git diff: working directory, a difference between commits

Switching between different branches
git checkout master: switch to master branch
git checkout dev: switch to the dev branch

by default, a remote repository is named as an origin by Git.
git push origin dev: push dev branch to the remote by the following command:

Removing a file or folder
git rm x.text
git commit -m "removing a file"
git push origin dev: don't forget to update remote

Git Merging
as standing on the master and merging changes from dev branch in
git checkout master
git merge dev
git add .  : add all changes from the dev branch
git commit -m "changes from dev"
git push origin master: commit local master into remote

When conflicts happen
git merge -- abort: get back the state before merge conflicts

editing confliction; OR:  choose

Alternatively, you can tell Git that you'll simply go with one of the edited versions, called "ours" or "theirs".

git checkout -- ours path/to/conflict-file.css

git checkout -- .  : git discard all changes that are not staged for commit

merger from a feature branch into local master

git checkout master

git pull

git checkout feature

git merge master

Git Revert
git revert undoes a specific commit and then creating a new commit.

git revert by default pops up a vi editor to modify the commit message; press ESC to command line, ZZ(Capital) command to save the file.

git revert specific-commit --n
git revert specific-commit -- no-edit

git log --oneline: it shows the commit ref(s)
git reflog

git revert commit-ref: then modifying the commit message; normally using the default text revert xxx

Press ESC, shift zz; then quit from the vi editor.
git log: it shows a revert commit created.

Git Reset

git reset: having three types of invocations. --soft(the commit tree head), --mixed(stagging index), or --hard(working directory).

Git/Linux Command
ps -ef|grep git: checking git process running
rm -f .git/index.lock : remove index locker file located in the ./git folder

Git Remove remote folder

making sure you are at the right branch

git checkout branch-expected

git rm --cached -rf ./.idea; -rf:  recursive and force

git commit -m "rm remote folder"

git push 

Git Remove remote and local folder

#remove a file

git rm password.text

#remove a folder

git rm -rf ./.idea

Config git

It allows to config git user name and email.

git config --global user.name "my-user-name"

git config --global user.email "my-email"

Git ignoring

Git disconnect remote repository

git remote -v: display all remote repositories linking to local repo.

git remote add remote_repo_name another_remote_git_repo_url: pointing to local repo. to another remote repo.

git remote rm remote_repo_name: removing the link to the remote repo, not rm remote repo.