Shell Basic Commands

Here String

diff <(echo "string1") <(echo "string2")

<(echo "string") construct creates a temporary file-like object that contains the string.

So in this way, we can walk around the file as input requirement of diff command.

Checking the exit status of ANY command in a pipeline

It's a pretty common thing in a shell script to want to check the exit status of the previous command. You can do this with the $? variable, as is widely known:

Command X echo $?

What gets difficult is when you execute a pipeline:

What you get is the result of the tee command, which writes the results to the display as well as to the results.txt file.

Command X | tee result.txt echo $?(This will return the result of tee result.txt)

To find out what grep returned, $? is of no use.

Instead, use the "${PIPESTATUS[]}" array variable. "${PIPESTATUS[0]}" tells us what grep returned, while "${PIPESTATUS[1]}" tells us what tee returned.

Command X | tee result.txt echo "${PIPESTATUS[0]}" (This will return the result of Command X)

OR

Command X | tee result.txt RC=("${PIPESTATUS[@]}")

Test command

test command or [ ] (which is equivalent to test)

Echo(Don't Interpret Newline Characters)

On macos, echo will interpret \n as a new line implicitly.

However, in CircleCI echo will not interpret \n implicitly. To interpret \n as a new line, we should use echo -e. To not interpret \n as a new line explicitly, we should use echo -E.

SSH/SCP with Jumphost

These only work when A can connect to B implicitly.

scp -J username@B local/path username@C:/remote/path

scp -i privateKeyOfC -J username@B local/path username@C:/remote/path

ssh -J username@B username@C

If A requires private key to connect to B, use the command below:

scp -oProxyCommand="ssh -i privateKeyOfB -W %h:%p userB@hostB" local/path userC@hostC:remote/path

ssh -oProxyCommand="ssh -i privateKeyOfB -W %h:%p userB@hostB" userC@hostC "ls"

Which private key is used for this command?

ssh -i hostC.id_rsa -oProxyCommand="ssh -i hostB.id_rsa -W %h:%p [email protected]" [email protected]

• hostC.id_rsa is stored on host A.
• hostB.id_rsa is stored on host A.
• No auto SSH connection to host C need to be set up on host B.
• No hostC.id_rsa need to be stored on host B.

ssh-keyscan

To add the fingerprint of a host manually:

ssh-keyscan -H DEST_HOSTNAME >> ~/.ssh/known_hosts

How ssh-keyscan Works:

Scanning Hosts: You provide ssh-keyscan with a list of hostnames or IP addresses. It then connects to the SSH port of each host (default is port 22).

Collecting Public Keys: For each host, ssh-keyscan retrieves the public key that the host presents during the SSH handshake process. This is the same key that a SSH client would see the first time it connects to the host.

Output: The utility then outputs the collected keys, typically in a format that can be directly added to an SSH known hosts file.

To resolve the hostname of a private instance, can I obtain the fingerprint from a third-party instance which can connect to the private instance, and then copy the fingerprint to an outside instance?

Yes, you can. But the weird thing is, this can solve the unknown hosts issue on my local machine, but not in the Github Action.

On my local machine, if I copy the fingerprint into the known_hosts, then there won't be warning anymore The authenticity of host 'ec2-XX.XX.XX.XX.us-west-2.compute.amazonaws.com (<no hostip for proxy command>)' can't be established. No matter if the instance is public or private.

But in the Github Action, it is not working if the instance is private, only works for the public instance. As for the private instance, there is still Host key verification failed. error.

So how to solve the Host Key Verification Failed issue when I want to connect to a private instance from Github Action?

If you are running in certain remote/scripting situations where you lack interactive access to the prompt-to-add-hostkey(the default ssh config StrictHostKeyChecking=ask), work around it like this:

Copy the fingerprint of hostB and hostC to the known_hosts file and use -o StrictHostKeyChecking=accept-new.

https://man7.org/linux/man-pages/man5/ssh_config.5.html

What is a host fingerprint?

The fingerprint is based on the host's public key, usually based on the /etc/ssh/ssh_host_rsa_key.pub file. Generally it's for easy identification/verification of the host you are connecting to.

Representation of the Public Key: The fingerprint is a shorter, more easily readable representation of a machine's public SSH key. It is created by applying a cryptographic hash function to the public key.

Unique Identifier: Just like the public key, its fingerprint is unique to each key (under normal cryptographic assumptions). This uniqueness allows it to reliably represent the public key for verification purposes.

Purpose: The main purpose of a fingerprint is to provide a convenient way for users and system administrators to verify the identity of a host. It's much easier to compare and confirm fingerprints, which are short strings, than to compare the full public keys, which are long and complex.

Security Checks: When you connect to an SSH server for the first time, the SSH client will display the fingerprint of the server's public key. You can then verify this fingerprint by comparing it to a trusted source (like a list provided by your organization or a verification call to a system admin). This step is crucial for ensuring that you are connecting to the legitimate server and not a malicious impersonator (to avoid man-in-the-middle attacks).

Changes in Fingerprint: If the public key of a host changes (due to key regeneration or for other reasons), its fingerprint will also change. This change will be evident when you next try to connect to the host, and your SSH client will warn you about this (as it could potentially indicate a security issue).

install command

The install command in Unix and Linux systems is quite versatile and is used for more than just creating directories. Its primary function is to copy files and set file attributes in the process. Here's a summary of its key functionalities:

Copying Files: install can be used to copy files from one location to another. This is similar to the cp command, but with additional capabilities.

Setting Permissions: When copying files, install allows you to set the permissions of the target file directly. This is a significant feature because, with standard copy commands, the copied file inherits the permissions of the original file, and you might need a separate command (like chmod) to change permissions.

Setting Ownership: install can also set the owner and group of the file being copied. This is similar to what you might do with the chown command, but install does it in the same step as copying the file.

Creating Directories: As you've seen, install can create directories with specific permissions, which is a feature not available in the basic mkdir command.

Stripping Debugging Symbols: When installing executables, install can strip debugging symbols from the file. This reduces the size of the installed binaries, which is often desirable in a production environment.

Preserving Timestamps: The command can preserve or set the timestamps of files when they are installed. This can be important for certain applications where file timestamps need to be maintained.

Use in Makefiles: install is particularly popular in makefiles for compiling and installing software. It automates the process of copying compiled binaries, scripts, and other files to their designated directories with the appropriate permissions and ownership.

. command

. is exactly the same as source command.

redirecting permission issue

command X | cat > The file you don't have permission will raise an error -bash: /XXX: Permission denied

However, sudo may not work as you expected.

command X | sudo cat > The file you don't have permission will raise the same error -bash: /XXX: Permission denied

When you use a redirection with sudo, the redirection is performed by your shell, not by sudo. Since your shell doesn't have permission to write to the file, the redirection would fail. tee handles this by running with sudo, thus having the necessary permissions to write to the file.

command X | sudo tee THE_FILE > /dev/null this will work with root permission and doesn't show any output.