Linux/UNIX Awk Command Tutorial with Examples

Awk is a scripting language which is used  for  processing or analyzing text files. Or we can say that awk command is mainly used for grouping of data based on either a column or field , or on a set of columns. Mainly it’s used for reporting data in a useful manner. It also employs Begin and End Blocks to process the data.

AWK  Stands for ‘Aho, Weinberger, and Kernighan‘

In this tutorial, we will learn awk command with practical examples.

Syntax of awk

# awk ‘pattern {action}’ input-file > output-file

Let’s take a input file with the following data

$ cat  awk_file
Name,Marks,Max Marks
Ram,200,1000
Shyam,500,1000
Ghyansham,1000
Abharam,800,1000
Hari,600,1000
Ram,400,1000

Now, let’s deep dive into practical examples of awk command.

1) Print all the lines from a file

By default, awk prints all lines of a file , so to print every line of above created file use below command :

$ awk '{print;}' awk_file
Name,Marks,Max Marks
Ram,200,1000
Shyam,500,1000
Ghyansham,1000
Abharam,800,1000
Hari,600,1000
Ram,400,1000

Note: In awk command ‘{print;}’ is used print all fields along with their values.

2) Print only specific field like 2nd & 3rd

In awk command, we use $ (dollar) symbol followed by field number to prints field values. In below example, we are printing field 2 (i.e Marks) and field 3 (i.e Max Marks)

$ awk -F "," '{print $2, $3;}' awk_file
Marks Max Marks
200 1000
500 1000
1000
800 1000
600 1000
400 1000

In the above command we have used the option  -F “,”  which specifies that comma (,) is the field separator in the file.

3) Print the lines which matches the pattern

I want to print the lines which contains the word “Hari & Ram”, run

$ awk '/Hari|Ram/' awk_file
Ram,200,1000
Hari,600,1000
Ram,400,1000

4) How do we find unique values in the first column of name

To print unique values from the first column, run below awk command

$ awk -F, '{a[$1];}END{for (i in a)print i;}' awk_file
Abharam
Hari
Name
Ghyansham
Ram
Shyam

5)  How to find the sum of data entry in a particular column

In awk command, it is also possible to perform some arithmetic operation based on search, syntax is shown below

$ awk -F, ‘$1==”Item1″{x+=$2;}END{print x}’ awk_file

In the below example, we search for Ram and then we add values of 2nd field for Ram word.

$ awk -F, '$1=="Ram"{x+=$2;}END{print x}' awk_file
600

6)  How to find the total of all numbers in a column

In awk command, we can also calculate the sum of all numbers in a column of a file. In the below example we are calculating the sum of all numbers of 2nd and 3rd column.

$ awk -F"," '{x+=$2}END{print x}' awk_file
3500
$ awk -F"," '{x+=$3}END{print x}' awk_file
5000

7)  How to find the sum of individual group records

For example, if we consider the first column than we can do the summation for the first column based on the items

$ awk -F, '{a[$1]+=$2;}END{for(i in a)print i", "a[i];}' awk_file
Abharam, 800
Hari, 600
Name, 0
Ghyansham, 1000
Ram, 600
Shyam, 500

8) Find the sum of all entries of specific columns and append it to the end of the file

As we already discuss that awk command can do sum of all numbers of a column, so to append the sum of column 2 and column 3 at the end of file, run

$ awk -F"," '{x+=$2;y+=$3;print}END{print "Total,"x,y}' awk_file
Name,Marks,Max Marks
Ram,200,1000
Shyam,500,1000
Ghyansham,1000
Abharam,800,1000
Hari,600,1000
Ram,400,1000
Total,3500 5000

9) How to find the count of entries against every column based on the first column

$ awk -F, '{a[$1]++;}END{for (i in a)print i, a[i];}' awk_file
Abharam 1
Hari 1
Name 1
Ghyansham 1
Ram 2
Shyam 1

10) How to print only the first record of every group

To print only first of every group, run below awk command

$ awk -F, '!a[$1]++' awk_file
Name,Marks,Max Marks
Ram,200,1000
Shyam,500,1000
Ghyansham,1000
Abharam,800,1000
Hari,600,1000

AWK Begin Block

Syntax for BEGIN block is

$ awk ‘BEGIN{awk initializing code}{actual AWK code}’   File-Name

Let us create a datafile with below contents

11)  How to populate each column names along with their corresponding data

$ awk 'BEGIN{print "Names\ttotal\tPPT\tDoc\txls"}{printf "%-s\t%d\t%d\t%d\t%d\n", $1,$2,$3,$4,$5}' datafile

12) How to change the field separator

As we can see space is the field separator in the datafile , in the below example we will change field separator  from space to “|”

$ awk 'BEGIN{OFS="|"}{print $1,$2,$3,$4,$5}' datafile

That’s all from this tutorial, I hope you found it informative. Please do share your feedback and queries in below comment’s section.

Recommended Read: 10 Quick Linux Tail Command with Examples