Let's compile a list of canonical Q&As

Question

A few of us were talking about this the other day in chat, as we are now quite a bit larger and with several thousand Q&As, there are certain questions that are asked very often. Things like redirecting output, process substitution, batch renaming of files, fixing/reinstalling grub, etc.

Let's compile a list of the best/most complete answers to such common questions here, and then we can always close new questions as duplicates of those. That way, we can have a single, canonical answer to these things which we can edit and make as good as we can and avoid having the dupes pointing at each other all over the place. The inspiration comes from this meta AU question:

Ideally, I would like to see an answer below that collects as many as possible. I will make a start but please edit to add more. As part of this initiative, we should also consider asking such questions and answering them ourselves. For example, I just read this question over on SU, having that kind of thing here seems like a very good idea (not specifically for hardware, although that would be useful, but that type of canonical question^*).

So, the idea would be to hash it out here, and once we have a decent list, make a meta Q&A like the ones above, listing them. Then, we make sure to close as dupes of the canonical.

--

Please vote for the answers below, upvote them if you consider the suggested Q&A to be a good candidate for the canonical one and downvote them if you don't. Upvoting the question itself is nice but less useful.

---

^* I made a start here: How can I replace a string in a file(s)?

Answer 1

Compiled list of questions

This list is compiled mostly from the answers below and some of my own, and is intended to be a quick reference for these questions.

shells

• where to set environmental variables
• How to permanently set environmental variables
• script runs from command line but not cron
• why you must quote - Gilles
• why you must quote - Stephane
• Why the “-” in the “#! /bin/sh -” shebang?
• #!/usr/bin/env bash vs #!/bin/bash
• `...` vs $(...)
• Difference between Login Shell and Non-Login Shell?
• Get exit status of process that's piped to another
• Difference Between: Terminal, Shell, TTY, and Console

sh

• shell arithmetic
• redirecting output
• handling filenames with unusual characters
• How to get the pid of the last executed command in shell script?
• How can I test if a variable is empty or contains only spaces?
• What are the shell's control and redirection operators?
• What is the “eval” command in bash?

bash

• how to show bash history
• How can I test if a variable is empty or contains only spaces?
• How to correctly add a path to PATH?
• In Bash, when to alias, when to script, and when to write a function?
• Why doesn't my Bash script recognize aliases?
• How can I get the size of a file in a bash script?

command line

• du is different than df
• finding the path of an executable
• batch renaming files
• getting system hardware info
• copy and paste on command line
• Delete a file starting with -
• Difference Between: Terminal, Shell, TTY, and Console
• How do I set an environment variable on the command line and have it appear in commands?
• How do I get the size of a directory on the command line?
• How to switch between users on one terminal?
• grep for multiple patterns
• How do I zip/unzip on the unix command line?
• Meaning of --
• How to set default file permissions for all folders/files in a directory?
• find - exec rm vs -delete
• Why is looping over find's output bad practice?

searching filesystems

• locate vs find: usage, pros and cons of each other
• Locating files and displaying their owner
• How to find inode number using “find” command?
• Getting all files that have been modified on a specific date
• List all files containing a specific string
• Script to find files with a specific extension but NOT containing specific text
• How to search for video files on Ubuntu?
• Checking if a file exists in several directories
• Find and move directories based on file type and date
• Shell script for moving oldest files?

text processing

• find and replace
• working with CSV files
• Append lines to file
• How to escape sed
• How can I use sed to replace a multi-line string?
• Why is using a shell loop to process text considered bad practice?
• How to add a newline to the end of a file?
• Windows/DOS text file, ^M, carriage returns: What is `^M` and how do I get rid of it? and/or How to test whether a file uses CRLF or LF without modifying it?

regex

• different types of regex

history

• Is Linux a Unix?
• Where did the “wheel” group get its name?
• What exactly is POSIX?

other

• installing new software on old OS
• fixing/reinstalling grub
• monitoring your server
• understanding op-mode
• passwordless machine access
• installing a custom kernel
• running 32 bit programs on 64 bit systems
• Why was '~' chosen to represent the home directory?

Answer 2

Where's my disk space?

Why do du and df return different values?

We seem to generally close these Qs as dupes of this one:

Why is there a discrepancy in disk usage reported by df and du?

For example these:

• `du -sh` reports different used size than `df -h`
• df vs. du: why so much difference?

I suggest we close as dupes of this one instead, I feel the answers here are better:

df says I have 20G more disk space used than du. Why?

---

UPDATE: @Gilles has posted The answer, so let's use that one:

Why are there so many different ways to measure disk usage?

Answer 3

Determining the path of an executable

Why not use "which"? What to use then?

Answer 4

Search for files satisfying a specific criterion. E.g. file type.

• locate vs find: usage, pros and cons of each other
• Locating files and displaying their owner
• How to find inode number using “find” command?
• Getting all files that have been modified on a specific date
• List all files containing a specific string
• Script to find files with a specific extension but NOT containing specific text
• How to search for video files on Ubuntu?
• Checking if a file exists in several directories
• Find and move directories based on file type and date
• Shell script for moving oldest files?

Answer 5

Redirecting output

How to redirect the output of any command?

Answer 6

Installing/Reinstalling/Fixing grub

How can I fix/install/reinstall grub?

I just edited my answer to make it more generic but it might be a good idea to modify the title if we're going to use this one.

Answer 7

Arithmetic calculations in the shell

How to do integer & float calculations, in bash or other languages/frameworks?

Answer 8

Batch renaming of files

Batch renaming files

Answer 9

Where should I set environment variables?

Is there a ".bashrc" equivalent file read by all shells?

Answer 10

Transform/Extract info from a set of CSV files

Is there a robust command line tool for processing csv files?

An additional question/answer pair could be created. The answer(s) should include regex/sed, awk based for those simpler CSVs, as well as Perl/Python based for when simple does not work.

A few good small example CSVs in the text that clearly illustrate where simple processing will just stop.

Answer 11

Searching and replacing strings in files.

There are a fair number of questions asked about this with variations, and it would be good if there was a meta answer pointing to the different variations.

Search and replacing a string on specific file extensions

Replacing string based on line number

Recursively replace string in files

Search and replace full line in recursive files

Putative general Q&A:

How can I replace a string in a file(s)?

Please add others.

Answer 12

Remember to quote your variables

Proposed question and answer:

• Security implications of forgetting to quote a variable in bash/POSIX shells

Answer 13

How do I monitor my server?

@slm's answer on this question is very good and inclusive:

Commands for determining level of usage of server

Answer 14

Using packages from more recent versions of a OS on earlier versions of that OS.

This question is probably most relevant on binary-based OSs like Debian/Fedora, and less relevant for source-based OSs like Gentoo/FreeBSD.

In these cases, one could give some hints about backporting, perhaps? For Debian, talk about backports, for ubuntu, PPAs, and so forth. This discussion should probably be restricted to official repositories, though.

I've posted this here: How can I install packages from a more recent version of my OS than the one I am running?.

Answer 15

See the output/control of a running command in another terminal

https://unix.stackexchange.com/search?q=reptyr

Answer 16

How does perl regexp and bash pattern work ?

This would need a basic description of different specific symbols and would need to explain the differences between the two - mode pattern vs perl regexp. This question could be linked with tag related to perl/sed/string replacement...
That's one of the most important things to master as it is used in almost any command line.

Answer 17

My script runs from the commandline, but not from cron

I was sure I had answered this before but could not find it or another generic answer (but I probably overlooked this).

Not able to run shell script using cron

Answer 18

Handling filenames with space and unusual characters

So many answers can be simplified and easier understood, when these edge cases are omitted. So the question Why does my shell script choke on whitespace or other special characters? provides a canonical method to handle filenames with uncommon characters. #()%^*space* *newline* [];'@:<>,.

Filenames to test your shell script against

How can I test my shell script's file-handling robustness?

Answer 19

Find Hardware Information of my machine from command line

• How can I have a linux box capabilities?
• 32-bit, 64-bit CPU op-mode on Linux

Answer 20

How to setup passwordless access to my machine?

This answer provides some of the available options.

script to automate scp in network

Answer 21

How to stop an application from buffering

https://unix.stackexchange.com/search?q=stdbuf

Answer 22

Install 32bit packages on a 64bit system

Proposed canonical:

How do I run 32-bit programs on a 64-bit Debian/Ubuntu?

Gilles' answer is excellent and distro agnostic and warlock's answer mentions multiarch.

Answer 23

How do I solve an APT dependency issue?

What information do I need to solve an APT dependency issue?

This one comes up fairly often. Most of the time, people have no idea what to do. See also "How do I resolve unmet dependencies?" on Ask Ubuntu. Essentially all the answers to this question are notable for totally missing the point.

Answer 24

Extract info from a XML/HTML file.

The answer(s) should include regex/sed, awk based for those simpler structured files and extractions, as well as Perl/Python/Ruby based for when simple does not work.

A few good small example segments in the text that clearly illustrate where simple processing will just stop.

Answer 25

How can I copy/paste files via the command line?

Proposed canonical:

• Copy and paste a file/directory from command line

Answer 26

Do not parse the output of ls

Proposed question and answer:

• Why not parse ls?

Answer 27

Common homework questions:

• Computing GCSE Controlled Assessment - Evaluate the usefulness of the top command and discuss how a computer technician might use it together with ps and kill in order to investigate a system that is misbehaving.

• Computing GCSE Controlled Assessment - Show and explain the content of the file that contains the Bash history

• Computing GCSE Controlled Assessment - At the command prompt, type ls -a. Demonstrate and explain what happens. Modify this command so that it only shows hidden files beginning with the letter ‘b’.

Answer 28

How can I install a custom kernel?

Configuring, compiling and installing a custom Linux kernel

[Extra text added to stop the system from converting "trivial" answers to comments]

Answer 29

How can I repeat the previous N commands in my shell?

Proposed canonical:

How to execute consecutive commands from history?

Answer 30

How can I remap the keys on my keyboard to other keys (or key combinations)?

Proposed canonicals:

• How to remap the Context Menu key to the Super key
• Use M-C, M-V, M-A for Copy, Paste, Select All
• how to assign another modifier to Alt key for X11?