Sed and awk: the stream editor and the field machine

🇷🇺 Русская версия

sed and awk are the two tools you reach for when grep is not enough and writing a script is too much. Both read a stream line by line, but they look at a line through a different lens: sed sees text to edit, awk sees columns to compute with. This is a tour of the flags and options that cover the vast majority of everyday use.

sed and awk: the flags you actually use — open the lesson page

Two tools, one pipeline

sed (stream editor) rewrites text line by line — a file, or everything that arrives through a pipe.

awk is really a small programming language for tabular data: it treats each line as a record split into fields, with variables, conditions, arrays and printf.

They compose well with everything else on the command line: grep finds lines, sed reshapes them, awk extracts and computes, sort/uniq order and count.

sed: the flags you will actually use

Option	Meaning
`-n`	do not print by default; print only what a `p` command asks for
`-e <script>`	add a script; repeat to run several in order
`-E` / `-r`	extended regex: `+`, `?`, `\|`, `()` without backslashes
`-i[.suffix]`	edit files in place; with a suffix, keep a backup
`-f <file>`	read the script from a file
`--`	end of options (editing a file that starts with `-`)

Substitute: `s/pattern/replacement/flags`

The workhorse. Common flags: g (every match, not just the first), i (case-insensitive), p (print the line if a substitution happened, paired with -n), and a number (2) to hit only the n-th match. & stands for the whole match, \1…\9 are capture groups.

sed 's/old/new/' file            # first match on each line
sed 's/old/new/g' file           # every match
sed -E 's/(.*)@(.*)/\2:\1/' file # swap around the @
sed 's/[[:space:]]\+/_/g' file   # spaces -> underscores

Print, delete, addresses

A command can be prefixed by an address: a line number, $ (last line), or a /regex/, or a range 2,5, /start/,/end/.

sed -n '10,20p' file     # lines 10 to 20
sed -n '/error/p' file   # only matching lines
sed '/^#/d' file         # drop comments
sed 's/foo/bar/; /baz/d' file   # several commands, one script

p prints, d deletes, s substitutes, q quits early, = prints the line number, a/i/c append/insert/change whole lines.

Editing files in place

sed -i 's/8080/9090/' config.txt      # rewrite the file
sed -i.bak 's/demo/prod/' config.txt  # keep config.txt.bak

On macOS the same option needs an explicit suffix: sed -i '' 's/a/b/' file.

awk: the flags you will actually use

Option	Meaning
`-F <sep>`	input field separator (default: runs of whitespace)
`-v name=value`	set a variable before the program runs
`-f <file>`	read the program from a file
`--`	end of options

A program is a list of pattern { action } rules. The pattern selects lines, the action runs on them. BEGIN runs before the first line, END after the last.

Fields and built-in variables

Each line is $0; $1, $2, … are the fields and $NF is the last one. NF is the field count, NR the line number overall, FNR the line number in the current file, FS/OFS the input/output separators.

awk '{print $1}' access.log            # first column
awk '{print $NF}' access.log           # last column
awk -F: '{print $1, $3}' /etc/passwd   # custom separator
awk -v limit=500 '$4 >= limit' access.log

Patterns and conditions

awk '$4 >= 500' access.log              # filter by a field
awk '/POST/' access.log                 # filter by a regex
awk 'NR > 1' file                       # skip the header
awk '$1 == "root", $1 == "daemon"' file # a range of records

BEGIN, END and arithmetic

awk 'BEGIN {print "ip count"} {c[$1]++} END {for (k in c) print k, c[k]}' access.log
awk '{sum += $4} END {print "total", sum}' access.log
awk 'END {print NR " lines"}' file

Reporting with printf

awk '{printf "%-10s %s\n", $1, $3}' access.log

%s, %d, %f, width and - (left align) behave like C's printf. Handy for turning a log into a table.

A few recipes worth stealing

# sum the sizes of the files in the current directory
ls -l | awk '{s += $5} END {print s}'

# unique IPs, most frequent first
awk '{print $1}' access.log | sort | uniq -c | sort -rn

# make a CSV out of whitespace-separated output
awk 'BEGIN {OFS=","} {print $1, $2, $3}' data.txt

# delete blank lines, squeeze spaces, trim the ends
sed -e '/^$/d' -e 's/  */ /g' -e 's/^ //; s/ $//' file

# rename every *.jpeg to *.jpg in the current directory
for f in *.jpeg; do mv "$f" "$(echo "$f" | sed 's/\.jpeg$/.jpg/')"; done

sed or awk?

Use sed for line-level text surgery: replace, delete, insert, rewrite a stream or a file in place. Use awk when a line is a record of fields and you need to select, compute, aggregate or format. When neither fits, they are only one pipe apart.

Both have a -i and a -n that do not mean the same thing, both accept extended regex (sed -E, awk uses POSIX ERE by default), and both are on virtually every Unix-like system, which is why the recipes above travel well.