take your linux skills next level =- learn AWK by example. ++hackers toolkit

hopefully by now your living in Linux) you are going to need to learn to program and I suggest basic scripting to start. Believe it or not the Bourne Again SHell (bASH) is very powerful and I will list a few of the common tricks and commands i usually pipe together and and up redirecting.

Lets get to the point — AWK
The primary discussion for this white-paper is to either familiarize users new to the world of awk, (which are almost guaranteed to already be installed on your target machine(s)). This is the key reason that I am reiterating the ninja f00 and absolute necessity of being able to parse any set of data, delimited with any character, or groups of white space, \r\n. you get the picture.

When I am especially initially gaining access to a target organization — and get on the first shell box — here is what im thinking.
#1 — anytime you make a connection to a target host/network even with vpn, tor, proxys the reason should be good and the plan should minimize overall interaction with the target — chances are if you for instance just hacked uslibrary.edu or one of the load balancers or reverse proxies in between, have your tools ready to rock. the goal is to identify any network chatter (if possible) of interest which may give us information about other VLANs, broadcast addresses, which routing protocol is in use, you want to verify the gateway, you want to do a masscan or zmap and a netdiscover and exit

\Linux tool ‘awk’ — It should be on almost every box imaginable

From ‘man awk’ page’: (note gawk and awk have slight diff)
DESCRIPTION

awk is the GNU Project’s implementation of the AWK programming language. It conforms to the definition of the language in the POSIX 1003.1 Standard. This version in turn is based on the description in The AWK
Programming Language, by Aho, Kernighan, and Weinberger, with the additional features found in the System
Release 4 version of UNIX awk

In simple terms, we will use awk to filter data and convert it into actionable information. Perfect example of usage is to extract meaningful information from logs of servers, network devices etc.

I am not a pro at text processing and I don’t want to claim to be. Let’s start out with what sED and AWK are. Both have been around since (probably) the early 70’s and the primary reason i bring it up here is I’ve been posting guides on pen testing and securing. You can almost bet one thing if you break into a box. It **will** have awk installed. What does this mean to you, and how does it benefit our “stick and move:” hacking approach (in and out is best, well planned, and wipe all logs). If you have been using *NIX (Remember if you see the term *NIX — it means Linux / UNIX some flavor. Text processing (and fast) can be an essential skill when getting on a new / target machine and mg assumptions are you would want to parse for instance /var/log/ for all IP addresses. Another Example, say we have a file data.dat. We will get to that in a moment.

Typically we could use “cut” or something else to easily parse the file, but since there are no clear delimiters (like a comma) it can get a little tricky. Here is where AWK comes in:

WHAT CAN WE DO WITH AWK?

1. AWK Operations:
   (a) Scans a file line by line
   (b) Splits each input line into fields
   © Compares input line/fields to pattern
   (d) Performs action(s) on matched lines

Useful For:
(a) Transform data files
(b) Produce formatted reports

Into some of the specifics so you can start mastering this essential tool!

-f program-file : Reads the AWK program source from the file 
                  program-file, instead of from the 
                  first command line argument.
-F fs            : Use fs for the input field separator

Let try something close to what i described earlier. Populate a textfile:
<If you havent populated a file with this method of using cat its good to know> type the following:

conditional and looping statements.

Conditional Statements

• if
• switch

Loop Statements

• while
• do-while
• for

bash$ cat > data.dat
mark manager account 45000
willsunil clerk account 25000
pauly manager sales 50000
amite manager account 47000
john peon sales 15000
meep clerk sales 2300
# at this point hit ctrl d (or ctrl c)

bash$ ls -altr data.dat}
# Check to see files size

• rw-r — r — 1 substance n1nja 153 May 31 06:32 test.txt
  YAY. WE CAN see we have populated the file using our cat method

** Core Commands
awk ‘{print}’ <ascii file. >:

Print the lines all of them in the file with this cmd.

bash$ awk ‘{print}’ test.txt
ajay manager account 45000
sunil clerk account 25000
varun manager sales 50000
amit manager account 47000
tarun peon sales 15000
deepak clerk sales 2390
bash$

Note: If you want an extra linefeed or two at the end of the data add
;echo;echo

Keep reading we haven’t even started the cool stuff:

Print the lines which match the given pattern.

$ awk '/manager/ {print}' data,dat
<Response>
ajay
varun
amit

We can see that AWK new to pull just the fields of the entries with the word: manager. note: the same goes for grep, egrep, and many commands in Linux, by putting the match in between // … Typically:

/subz/genius/ — Would replace subz with genius

So i hope you’re starting to see some value, let’s pick up the pace. To print certain fields (delimited and detected by AWK as whitespace)

$ awk ‘{print $1,$4}’ ./data.dat. #. Tell awk ro print 1st am 4th column
ajay 45000 # We only see field ONE and FOUR on stdout
sunil 25000 # We only see field ONE and FOUR on stdout
varun 50000 # We only see field ONE and FOUR on stdout
amit 47000 # We only see field ONE and FOUR on stdout
tarun 15000 # We only see field ONE and FOUR on stdout
deepak 2300 # We only see field ONE and FOUR on stdout

The truly amazing part about AWK and this text parsing is when you need to do complex output, such as rearranging fields on the fly, tallying fields, getting a ton of data manipulated how you want it. when compared to PERL (which is fast) AWK is much faster. We can easily parse 3 million lines.

These are the built in ‘variables’ or options that AWK understands:

Built-In Variables In Awk

Awk’s built-in variables include the field variables — $1, $2, $3, and so on ($0 is the entire line) — that break a line of text into individual words or pieces called fields.

• NR: NR command keeps a current count of the number of input records. Remember that records are usually lines. Awk command performs the pattern/action statements once for each record in a file. THIS YOU MUST REMEMBER ITS KEY
• NF: NF command keeps a count of the number of fields within the current input record.
• FS: FS command contains the field separator character which is used to divide fields on the input line. The default is “white space”, meaning space and tab characters. FS can be reassigned to another character (typically in BEGIN) to change the field separator.
• RS: RS command stores the current record separator character. Since, by default, an input line is the input record, the default record separator character is a newline.
• OFS: OFS command stores the output field separator, which separates the fields when Awk prints them. The default is a blank space. Whenever print has several parameters separated with commas, it will print the value of OFS in between each parameter.
• ORS: ORS command stores the output record separator, which separates the output lines when Awk prints them. The default is a newline character. print automatically outputs the contents of ORS at the end of whatever it is given to print.

Using AWK “NR” built-in variables (Display Line Number)

Note: REMEMBER THE SYNTAX this is key:
awk ‘{CMD}’ <filename>… lets try this example:

awk ‘{print $1,$NF}’ ./data.dat
ajay 45000
sunil 25000
varun 50000
amit 47000
tarun 15000
deepak 2300
— — We can see from above that we are seeing the first entry in the file as well as

Another use of NR built-in variables (Display Line From 3 to 6)

$ awk 'NR==3, NR==6 {print NR,$0}' employee.txt

Into the hacker stuff!

AWK is killer at Field Matching

Building specific and narrow search patterns can become critical when dealing with complex text files.

Lets take a look at parsing /var/log/apache2.

take your linux skills next level =- learn AWK by example. ++hackers toolkit

hopefully by now your living in Linux) you are going to need to learn to program and I suggest basic scripting to start. Believe it or not the Bourne Again SHell (bASH) is very powerful and I will list a few of the common tricks and commands i usually pipe together and and up redirecting.

Lets get to the point — AWK
The primary discussion for this white-paper is to either familiarize users new to the world of awk, (which are almost guaranteed to already be installed on your target machine(s)). This is the key reason that I am reiterating the ninja f00 and absolute necessity of being able to parse any set of data, delimited with any character, or groups of white space, \r\n. you get the picture.

When I am especially initially gaining access to a target organization — and get on the first shell box — here is what im thinking.
#1 — anytime you make a connection to a target host/network even with vpn, tor, proxys the reason should be good and the plan should minimize overall interaction with the target — chances are if you for instance just hacked uslibrary.edu or one of the load balancers or reverse proxies in between, have your tools ready to rock. the goal is to identify any network chatter (if possible) of interest which may give us information about other VLANs, broadcast addresses, which routing protocol is in use, you want to verify the gateway, you want to do a masscan or zmap and a netdiscover and exit

\Linux tool ‘awk’ — It should be on almost every box imaginable

From ‘man awk’ page’: (note gawk and awk have slight diff)
DESCRIPTION

awk is the GNU Project’s implementation of the AWK programming language. It conforms to the definition of the language in the POSIX 1003.1 Standard. This version in turn is based on the description in The AWK
Programming Language, by Aho, Kernighan, and Weinberger, with the additional features found in the System
Release 4 version of UNIX awk

In simple terms, we will use awk to filter data and convert it into actionable information. Perfect example of usage is to extract meaningful information from logs of servers, network devices etc.

I am not a pro at text processing and I don’t want to claim to be. Let’s start out with what sED and AWK are. Both have been around since (probably) the early 70’s and the primary reason i bring it up here is I’ve been posting guides on pen testing and securing. You can almost bet one thing if you break into a box. It **will** have awk installed. What does this mean to you, and how does it benefit our “stick and move:” hacking approach (in and out is best, well planned, and wipe all logs). If you have been using *NIX (Remember if you see the term *NIX — it means Linux / UNIX some flavor. Text processing (and fast) can be an essential skill when getting on a new / target machine and mg assumptions are you would want to parse for instance /var/log/ for all IP addresses. Another Example, say we have a file data.dat. We will get to that in a moment.

Typically we could use “cut” or something else to easily parse the file, but since there are no clear delimiters (like a comma) it can get a little tricky. Here is where AWK comes in:

WHAT CAN WE DO WITH AWK?

1. AWK Operations:
   (a) Scans a file line by line
   (b) Splits each input line into fields
   © Compares input line/fields to pattern
   (d) Performs action(s) on matched lines

Useful For:
(a) Transform data files
(b) Produce formatted reports

Into some of the specifics so you can start mastering this essential tool!

-f program-file : Reads the AWK program source from the file 
                  program-file, instead of from the 
                  first command line argument.
-F fs            : Use fs for the input field separator

Let try something close to what i described earlier. Populate a textfile:
<If you havent populated a file with this method of using cat its good to know> type the following:

conditional and looping statements.

Conditional Statements

• if
• switch

Loop Statements

• while
• do-while
• for

bash$ cat > data.dat
mark manager account 45000
willsunil clerk account 25000
pauly manager sales 50000
amite manager account 47000
john peon sales 15000
meep clerk sales 2300
# at this point hit ctrl d (or ctrl c)

bash$ ls -altr data.dat}
# Check to see files size

• rw-r — r — 1 substance n1nja 153 May 31 06:32 test.txt
  YAY. WE CAN see we have populated the file using our cat method

** Core Commands
awk ‘{print}’ <ascii file. >:

Print the lines all of them in the file with this cmd.

bash$ awk ‘{print}’ test.txt
ajay manager account 45000
sunil clerk account 25000
varun manager sales 50000
amit manager account 47000
tarun peon sales 15000
deepak clerk sales 2390
bash$

Note: If you want an extra linefeed or two at the end of the data add
;echo;echo

Keep reading we haven’t even started the cool stuff:

Print the lines which match the given pattern.

$ awk '/manager/ {print}' data,dat
<Response>
ajay
varun
amit

We can see that AWK new to pull just the fields of the entries with the word: manager. note: the same goes for grep, egrep, and many commands in Linux, by putting the match in between // … Typically:

/subz/genius/ — Would replace subz with genius

So i hope you’re starting to see some value, let’s pick up the pace. To print certain fields (delimited and detected by AWK as whitespace)

$ awk ‘{print $1,$4}’ ./data.dat. #. Tell awk ro print 1st am 4th column
ajay 45000 # We only see field ONE and FOUR on stdout
sunil 25000 # We only see field ONE and FOUR on stdout
varun 50000 # We only see field ONE and FOUR on stdout
amit 47000 # We only see field ONE and FOUR on stdout
tarun 15000 # We only see field ONE and FOUR on stdout
deepak 2300 # We only see field ONE and FOUR on stdout

The truly amazing part about AWK and this text parsing is when you need to do complex output, such as rearranging fields on the fly, tallying fields, getting a ton of data manipulated how you want it. when compared to PERL (which is fast) AWK is much faster. We can easily parse 3 million lines.

These are the built in ‘variables’ or options that AWK understands:

Built-In Variables In Awk

Awk’s built-in variables include the field variables — $1, $2, $3, and so on ($0 is the entire line) — that break a line of text into individual words or pieces called fields.

• NR: NR command keeps a current count of the number of input records. Remember that records are usually lines. Awk command performs the pattern/action statements once for each record in a file. THIS YOU MUST REMEMBER ITS KEY
• NF: NF command keeps a count of the number of fields within the current input record.
• FS: FS command contains the field separator character which is used to divide fields on the input line. The default is “white space”, meaning space and tab characters. FS can be reassigned to another character (typically in BEGIN) to change the field separator.
• RS: RS command stores the current record separator character. Since, by default, an input line is the input record, the default record separator character is a newline.
• OFS: OFS command stores the output field separator, which separates the fields when Awk prints them. The default is a blank space. Whenever print has several parameters separated with commas, it will print the value of OFS in between each parameter.
• ORS: ORS command stores the output record separator, which separates the output lines when Awk prints them. The default is a newline character. print automatically outputs the contents of ORS at the end of whatever it is given to print.

Using AWK “NR” built-in variables (Display Line Number)

Note: REMEMBER THE SYNTAX this is key:
awk ‘{CMD}’ <filename>… lets try this example:

awk ‘{print $1,$NF}’ ./data.dat
ajay 45000
sunil 25000
varun 50000
amit 47000
tarun 15000
deepak 2300
— — We can see from above that we are seeing the first entry in the file as well as