Go to the previous, next section.
In the typical awk program, all input is read either from the
standard input (usually the keyboard) or from files whose names you
specify on the awk command line. If you specify input files,
awk reads data from the first one until it reaches the end; then
it reads the second file until it reaches the end, and so on. The name
of the current input file can be found in the built-in variable
FILENAME (see section Built-in Variables).
The input is read in units called records, and processed by the rules one record at a time. By default, each record is one line. Each record read is split automatically into fields, to make it more convenient for a rule to work on parts of the record under consideration.
On rare occasions you will need to use the getline command,
which can do explicit input from any number of files (see section Explicit Input with getline).
The awk language divides its input into records and fields.
Records are separated by a character called the record separator.
By default, the record separator is the newline character. Therefore,
normally, a record is a line of text.
Sometimes you may want to use a different character to separate your
records. You can use different characters by changing the built-in
variable RS.
The value of RS is a string that says how to separate records;
the default value is "\n", the string of just a newline
character. This is why records are, by default, single lines.
RS can have any string as its value, but only the first character
of the string is used as the record separator. The other characters are
ignored. RS is exceptional in this regard; awk uses the
full value of all its other built-in variables.
You can change the value of RS in the awk program with the
assignment operator, `=' (see section Assignment Expressions). The new
record-separator character should be enclosed in quotation marks to make
a string constant. Often the right time to do this is at the beginning
of execution, before any input has been processed, so that the very
first record will be read with the proper separator. To do this, use
the special BEGIN pattern (see section BEGIN and END Special Patterns). For
example:
awk 'BEGIN { RS = "/" } ; { print $0 }' BBS-list
changes the value of RS to "/", before reading any input.
This is a string whose first character is a slash; as a result, records
are separated by slashes. Then the input file is read, and the second
rule in the awk program (the action with no pattern) prints each
record. Since each print statement adds a newline at the end of
its output, the effect of this awk program is to copy the input
with each slash changed to a newline.
Another way to change the record separator is on the command line,
using the variable-assignment feature (see section Invocation of awk).
awk '...' RS="/" source-file
This sets RS to `/' before processing source-file.
The empty string (a string of no characters) has a special meaning
as the value of RS: it means that records are separated only
by blank lines. See section Multiple-Line Records, for more details.
The awk utility keeps track of the number of records that have
been read so far from the current input file. This value is stored in a
built-in variable called FNR. It is reset to zero when a new
file is started. Another built-in variable, NR, is the total
number of input records read so far from all files. It starts at zero
but is never automatically reset to zero.
If you change the value of RS in the middle of an awk run,
the new value is used to delimit subsequent records, but the record
currently being processed (and records already finished) are not
affected.
When awk reads an input record, the record is
automatically separated or parsed by the interpreter into pieces
called fields. By default, fields are separated by whitespace,
like words in a line.
Whitespace in awk means any string of one or more spaces and/or
tabs; other characters such as newline, formfeed, and so on, that are
considered whitespace by other languages are not considered
whitespace by awk.
The purpose of fields is to make it more convenient for you to refer to
these pieces of the record. You don't have to use them--you can
operate on the whole record if you wish--but fields are what make
simple awk programs so powerful.
To refer to a field in an awk program, you use a dollar-sign,
`$', followed by the number of the field you want. Thus, $1
refers to the first field, $2 to the second, and so on. For
example, suppose the following is a line of input:
This seems like a pretty nice example.
Here the first field, or $1, is `This'; the second field, or
$2, is `seems'; and so on. Note that the last field,
$7, is `example.'. Because there is no space between the
`e' and the `.', the period is considered part of the seventh
field.
No matter how many fields there are, the last field in a record can be
represented by $NF. So, in the example above, $NF would
be the same as $7, which is `example.'. Why this works is
explained below (see section Non-constant Field Numbers). If you try to refer to a
field beyond the last one, such as $8 when the record has only 7
fields, you get the empty string.
Plain NF, with no `$', is a built-in variable whose value
is the number of fields in the current record.
$0, which looks like an attempt to refer to the zeroth field, is
a special case: it represents the whole input record. This is what you
would use when you aren't interested in fields.
Here are some more examples:
awk '$1 ~ /foo/ { print $0 }' BBS-list
This example prints each record in the file `BBS-list' whose first
field contains the string `foo'. The operator `~' is called a
matching operator (see section Comparison Expressions); it tests whether a
string (here, the field $1) contains a match for a given regular
expression.
By contrast, the following example:
awk '/foo/ { print $1, $NF }' BBS-list
looks for `foo' in the entire record and prints the first field and the last field for each input record containing a match.
The number of a field does not need to be a constant. Any expression in
the awk language can be used after a `$' to refer to a
field. The value of the expression specifies the field number. If the
value is a string, rather than a number, it is converted to a number.
Consider this example:
awk '{ print $NR }'
Recall that NR is the number of records read so far: 1 in the
first record, 2 in the second, etc. So this example prints the first
field of the first record, the second field of the second record, and so
on. For the twentieth record, field number 20 is printed; most likely,
the record has fewer than 20 fields, so this prints a blank line.
Here is another example of using expressions as field numbers:
awk '{ print $(2*2) }' BBS-list
The awk language must evaluate the expression (2*2) and use
its value as the number of the field to print. The `*' sign
represents multiplication, so the expression 2*2 evaluates to 4.
The parentheses are used so that the multiplication is done before the
`$' operation; they are necessary whenever there is a binary
operator in the field-number expression. This example, then, prints the
hours of operation (the fourth field) for every line of the file
`BBS-list'.
If the field number you compute is zero, you get the entire record.
Thus, $(2-2) has the same value as $0. Negative field
numbers are not allowed.
The number of fields in the current record is stored in the built-in
variable NF (see section Built-in Variables). The expression
$NF is not a special feature: it is the direct consequence of
evaluating NF and using its value as a field number.
You can change the contents of a field as seen by awk within an
awk program; this changes what awk perceives as the
current input record. (The actual input is untouched: awk never
modifies the input file.)
Look at this example:
awk '{ $3 = $2 - 10; print $2, $3 }' inventory-shipped
The `-' sign represents subtraction, so this program reassigns
field three, $3, to be the value of field two minus ten,
$2 - 10. (See section Arithmetic Operators.) Then field two, and the
new value for field three, are printed.
In order for this to work, the text in field $2 must make sense
as a number; the string of characters must be converted to a number in
order for the computer to do arithmetic on it. The number resulting
from the subtraction is converted back to a string of characters which
then becomes field three. See section Conversion of Strings and Numbers.
When you change the value of a field (as perceived by awk), the
text of the input record is recalculated to contain the new field where
the old one was. Therefore, $0 changes to reflect the altered
field. Thus,
awk '{ $2 = $2 - 10; print $0 }' inventory-shipped
prints a copy of the input file, with 10 subtracted from the second field of each line.
You can also assign contents to fields that are out of range. For example:
awk '{ $6 = ($5 + $4 + $3 + $2) ; print $6 }' inventory-shipped
We've just created $6, whose value is the sum of fields
$2, $3, $4, and $5. The `+' sign
represents addition. For the file `inventory-shipped', $6
represents the total number of parcels shipped for a particular month.
Creating a new field changes the internal awk copy of the current
input record--the value of $0. Thus, if you do `print $0'
after adding a field, the record printed includes the new field, with
the appropriate number of field separators between it and the previously
existing fields.
This recomputation affects and is affected by several features not yet
discussed, in particular, the output field separator, OFS,
which is used to separate the fields (see section Output Separators), and
NF (the number of fields; see section Examining Fields). For example, the
value of NF is set to the number of the highest field you
create.
Note, however, that merely referencing an out-of-range field
does not change the value of either $0 or NF.
Referencing an out-of-range field merely produces a null string. For
example:
if ($(NF+1) != "")
print "can't happen"
else
print "everything is normal"
should print `everything is normal', because NF+1 is certain
to be out of range. (See section The if Statement, for more information about
awk's if-else statements.)
The way awk splits an input record into fields is controlled by
the field separator, which is a single character or a regular
expression. awk scans the input record for matches for the
separator; the fields themselves are the text between the matches. For
example, if the field separator is `oo', then the following line:
moo goo gai pan
would be split into three fields: `m', ` g' and ` gai pan'.
The field separator is represented by the built-in variable FS.
Shell programmers take note! awk does not use the name
IFS which is used by the shell.
You can change the value of FS in the awk program with the
assignment operator, `=' (see section Assignment Expressions). Often the right
time to do this is at the beginning of execution, before any input has
been processed, so that the very first record will be read with the
proper separator. To do this, use the special BEGIN pattern
(see section BEGIN and END Special Patterns). For example, here we set the value of FS to
the string ",":
awk 'BEGIN { FS = "," } ; { print $2 }'
Given the input line,
John Q. Smith, 29 Oak St., Walamazoo, MI 42139
this awk program extracts the string `29 Oak St.'.
Sometimes your input data will contain separator characters that don't separate fields the way you thought they would. For instance, the person's name in the example we've been using might have a title or suffix attached, such as `John Q. Smith, LXIX'. From input containing such a name:
John Q. Smith, LXIX, 29 Oak St., Walamazoo, MI 42139
the previous sample program would extract `LXIX', instead of `29 Oak St.'. If you were expecting the program to print the address, you would be surprised. So choose your data layout and separator characters carefully to prevent such problems.
As you know, by default, fields are separated by whitespace sequences
(spaces and tabs), not by single spaces: two spaces in a row do not
delimit an empty field. The default value of the field separator is a
string " " containing a single space. If this value were
interpreted in the usual way, each space character would separate
fields, so two spaces in a row would make an empty field between them.
The reason this does not happen is that a single space as the value of
FS is a special case: it is taken to specify the default manner
of delimiting fields.
If FS is any other single character, such as ",", then
each occurrence of that character separates two fields. Two consecutive
occurrences delimit an empty field. If the character occurs at the
beginning or the end of the line, that too delimits an empty field. The
space character is the only single character which does not follow these
rules.
More generally, the value of FS may be a string containing any
regular expression. Then each match in the record for the regular
expression separates fields. For example, the assignment:
FS = ", \t"
makes every area of an input line that consists of a comma followed by a space and a tab, into a field separator. (`\t' stands for a tab.)
For a less trivial example of a regular expression, suppose you want
single spaces to separate fields the way single commas were used above.
You can set FS to "[ ]". This regular expression
matches a single space and nothing else.
FS can be set on the command line. You use the `-F' argument to
do so. For example:
awk -F, 'program' input-files
sets FS to be the `,' character. Notice that the argument uses
a capital `F'. Contrast this with `-f', which specifies a file
containing an awk program. Case is significant in command options:
the `-F' and `-f' options have nothing to do with each other.
You can use both options at the same time to set the FS argument
and get an awk program from a file.
As a special case, in compatibility mode (see section Invocation of awk), if the
argument to `-F' is `t', then FS is set to the tab
character. (This is because if you type `-F\t', without the quotes,
at the shell, the `\' gets deleted, so awk figures that you
really want your fields to be separated with tabs, and not `t's.
Use `FS="t"' on the command line if you really do want to separate
your fields with `t's.)
For example, let's use an awk program file called `baud.awk'
that contains the pattern /300/, and the action `print $1'.
Here is the program:
/300/ { print $1 }
Let's also set FS to be the `-' character, and run the
program on the file `BBS-list'. The following command prints a
list of the names of the bulletin boards that operate at 300 baud and
the first three digits of their phone numbers:
awk -F- -f baud.awk BBS-list
It produces this output:
aardvark 555 alpo barfly 555 bites 555 camelot 555 core 555 fooey 555 foot 555 macfoo 555 sdace 555 sabafoo 555
Note the second line of output. If you check the original file, you will see that the second line looked like this:
alpo-net 555-3412 2400/1200/300 A
The `-' as part of the system's name was used as the field separator, instead of the `-' in the phone number that was originally intended. This demonstrates why you have to be careful in choosing your field and record separators.
The following program searches the system password file, and prints the entries for users who have no password:
awk -F: '$2 == ""' /etc/passwd
Here we use the `-F' option on the command line to set the field separator. Note that fields in `/etc/passwd' are separated by colons. The second field represents a user's encrypted password, but if the field is empty, that user has no password.
In some data bases, a single line cannot conveniently hold all the information in one entry. In such cases, you can use multi-line records.
The first step in doing this is to choose your data format: when records are not defined as single lines, how do you want to define them? What should separate records?
One technique is to use an unusual character or string to separate
records. For example, you could use the formfeed character (written
`\f' in awk, as in C) to separate them, making each record
a page of the file. To do this, just set the variable RS to
"\f" (a string containing the formfeed character). Any
other character could equally well be used, as long as it won't be part
of the data in a record.
Another technique is to have blank lines separate records. By a special
dispensation, a null string as the value of RS indicates that
records are separated by one or more blank lines. If you set RS
to the null string, a record always ends at the first blank line
encountered. And the next record doesn't start until the first nonblank
line that follows--no matter how many blank lines appear in a row, they
are considered one record-separator.
The second step is to separate the fields in the record. One way to do
this is to put each field on a separate line: to do this, just set the
variable FS to the string "\n". (This simple regular
expression matches a single newline.)
Another idea is to divide each of the lines into fields in the normal
manner. This happens by default as a result of a special feature: when
RS is set to the null string, the newline character always
acts as a field separator. This is in addition to whatever field
separations result from FS.
The original motivation for this special exception was probably so that
you get useful behavior in the default case (i.e., FS == "
"). This feature can be a problem if you really don't want the
newline character to separate fields, since there is no way to
prevent it. However, you can work around this by using the split
function to break up the record manually (see section Built-in Functions for String Manipulation).
getline
So far we have been getting our input files from awk's main
input stream--either the standard input (usually your terminal) or the
files specified on the command line. The awk language has a
special built-in command called getline that
can be used to read input under your explicit control.
This command is quite complex and should not be used by
beginners. It is covered here because this is the chapter on input.
The examples that follow the explanation of the getline command
include material that has not been covered yet. Therefore, come back
and study the getline command after you have reviewed the
rest of this manual and have a good knowledge of how awk works.
getline returns 1 if it finds a record, and 0 if the end of the
file is encountered. If there is some error in getting a record, such
as a file that cannot be opened, then getline returns -1.
In the following examples, command stands for a string value that represents a shell command.
getline
getline command can be used without arguments to read input
from the current input file. All it does in this case is read the next
input record and split it up into fields. This is useful if you've
finished processing the current record, but you want to do some special
processing right now on the next record. Here's an
example:
awk '{
if (t = index($0, "/*")) {
if(t > 1)
tmp = substr($0, 1, t - 1)
else
tmp = ""
u = index(substr($0, t + 2), "*/")
while (! u) {
getline
t = -1
u = index($0, "*/")
}
if(u <= length($0) - 2)
$0 = tmp substr($0, t + u + 3)
else
$0 = tmp
}
print $0
}'
This awk program deletes all comments, `/* ...
*/', from the input. By replacing the `print $0' with other
statements, you could perform more complicated processing on the
decommented input, such as searching it for matches for a regular
expression.
This form of the getline command sets NF (the number of
fields; see section Examining Fields), NR (the number of records read so far;
see section How Input is Split into Records), FNR (the number of records read from this input
file), and the value of $0.
Note: the new value of $0 is used in testing
the patterns of any subsequent rules. The original value
of $0 that triggered the rule which executed getline
is lost. By contrast, the next statement reads a new record
but immediately begins processing it normally, starting with the first
rule in the program. See section The next Statement.
getline var
getline reads a record into the variable var.
This is useful when you want your program to read the next record from
the current input file, but you don't want to subject the record to the
normal input processing.
For example, suppose the next line is a comment, or a special string,
and you want to read it, but you must make certain that it won't trigger
any rules. This version of getline allows you to read that line
and store it in a variable so that the main
read-a-line-and-check-each-rule loop of awk never sees it.
The following example swaps every two lines of input. For example, given:
wan tew free phore
it outputs:
tew wan phore free
Here's the program:
awk '{
if ((getline tmp) > 0) {
print tmp
print $0
} else
print $0
}'
The getline function used in this way sets only the variables
NR and FNR (and of course, var). The record is not
split into fields, so the values of the fields (including $0) and
the value of NF do not change.
getline < file
getline function takes its input from the file
file. Here file is a string-valued expression that
specifies the file name. `< file' is called a redirection
since it directs input to come from a different place.
This form is useful if you want to read your input from a particular file, instead of from the main input stream. For example, the following program reads its input record from the file `foo.input' when it encounters a first field with a value equal to 10 in the current input file.
awk '{
if ($1 == 10) {
getline < "foo.input"
print
} else
print
}'
Since the main input stream is not used, the values of NR and
FNR are not changed. But the record read is split into fields in
the normal manner, so the values of $0 and other fields are
changed. So is the value of NF.
This does not cause the record to be tested against all the patterns
in the awk program, in the way that would happen if the record
were read normally by the main processing loop of awk. However
the new record is tested against any subsequent rules, just as when
getline is used without a redirection.
getline var < file
getline function takes its input from the file
file and puts it in the variable var. As above, file
is a string-valued expression that specifies the file to read from.
In this version of getline, none of the built-in variables are
changed, and the record is not split into fields. The only variable
changed is var.
For example, the following program copies all the input files to the output, except for records that say `@include filename'. Such a record is replaced by the contents of the file filename.
awk '{
if (NF == 2 && $1 == "@include") {
while ((getline line < $2) > 0)
print line
close($2)
} else
print
}'
Note here how the name of the extra input file is not built into the program; it is taken from the data, from the second field on the `@include' line.
The close function is called to ensure that if two identical
`@include' lines appear in the input, the entire specified file is
included twice. See section Closing Input Files and Pipes.
One deficiency of this program is that it does not process nested `@include' statements the way a true macro preprocessor would.
command | getline
getline. A pipe is
simply a way to link the output of one program to the input of another. In
this case, the string command is run as a shell command and its output
is piped into awk to be used as input. This form of getline
reads one record from the pipe.
For example, the following program copies input to output, except for lines that begin with `@execute', which are replaced by the output produced by running the rest of the line as a shell command:
awk '{
if ($1 == "@execute") {
tmp = substr($0, 10)
while ((tmp | getline) > 0)
print
close(tmp)
} else
print
}'
The close function is called to ensure that if two identical
`@execute' lines appear in the input, the command is run again for
each one. See section Closing Input Files and Pipes.
Given the input:
foo bar baz @execute who bletch
the program might produce:
foo bar baz hack ttyv0 Jul 13 14:22 hack ttyp0 Jul 13 14:23 (gnu:0) hack ttyp1 Jul 13 14:23 (gnu:0) hack ttyp2 Jul 13 14:23 (gnu:0) hack ttyp3 Jul 13 14:23 (gnu:0) bletch
Notice that this program ran the command who and printed the result.
(If you try this program yourself, you will get different results, showing
you logged in.)
This variation of getline splits the record into fields, sets the
value of NF and recomputes the value of $0. The values of
NR and FNR are not changed.
command | getline var
getline and into the variable var. For example, the
following program reads the current date and time into the variable
current_time, using the utility called date, and then
prints it.
awk 'BEGIN {
"date" | getline current_time
close("date")
print "Report printed on " current_time
}'
In this version of getline, none of the built-in variables are
changed, and the record is not split into fields.
If the same file name or the same shell command is used with
getline more than once during the execution of an awk
program, the file is opened (or the command is executed) only the first time.
At that time, the first record of input is read from that file or command.
The next time the same file or command is used in getline, another
record is read from it, and so on.
This implies that if you want to start reading the same file again from
the beginning, or if you want to rerun a shell command (rather that
reading more output from the command), you must take special steps.
What you can do is use the close function, as follows:
close(filename)
or
close(command)
The argument filename or command can be any expression. Its value must exactly equal the string that was used to open the file or start the command--for example, if you open a pipe with this:
"sort -r names" | getline foo
then you must close it with this:
close("sort -r names")
Once this function call is executed, the next getline from that
file or command will reopen the file or rerun the command.
Go to the previous, next section.