1
00:00:00,980 --> 00:00:04,820
A very useful data source is compiled files.

2
00:00:04,820 --> 00:00:07,380
This allows our pipeline to benefit

3
00:00:07,380 --> 00:00:11,620
from the sophisticated processing
performed by each language's compiler.

4
00:00:11,620 --> 00:00:13,860
Thus, we don't need to reinvent the wheel,

5
00:00:13,860 --> 00:00:17,500
by devising our own lexical analysis
and parsing tools.

6
00:00:17,500 --> 00:00:21,180
Instead, we can obtain
the processed source code data

7
00:00:21,180 --> 00:00:25,880
in the canonical form
in which the compiler digests it.

8
00:00:25,880 --> 00:00:30,440
Many types of data files
are amenable to this kind of processing.

9
00:00:30,440 --> 00:00:35,920
These include native code object files,
virtual machine bytecode files,

10
00:00:35,920 --> 00:00:37,700
linked executable files,

11
00:00:37,700 --> 00:00:41,340
as well as statically
and dynamically linked libraries.

12
00:00:41,340 --> 00:00:43,780
This approach works on many environments,

13
00:00:43,780 --> 00:00:47,460
for example,
for native code on Unix and Windows

14
00:00:47,460 --> 00:00:50,720
and in the Java ecosystem.

15
00:00:50,720 --> 00:00:54,160
We can see what's
in the native code object files

16
00:00:54,160 --> 00:00:58,340
with the nm command.

17
00:00:58,340 --> 00:01:01,720
Here's a very simple C program, called hello.c.

18
00:01:01,720 --> 00:01:03,100
It includes a header,

19
00:01:03,100 --> 00:01:05,160
it defines a global variable named global

20
00:01:05,160 --> 00:01:07,320
and a static variable named local,

21
00:01:07,320 --> 00:01:09,980
which is only visible within the file.

22
00:01:09,980 --> 00:01:12,480
There is also a local function named localfun,

23
00:01:12,480 --> 00:01:15,320
and a main function
which is globally visible.

24
00:01:15,320 --> 00:01:19,000
Main calls the fprintf function
to print on the standard output:

25
00:01:19,000 --> 00:01:20,479
"hello, world: ".

26
00:01:20,480 --> 00:01:24,540
Next to "hello, world: ",
it prints the output of localfun.

27
00:01:24,540 --> 00:01:26,860
I compile the file hello.c,

28
00:01:26,860 --> 00:01:30,580
with the cc command
which stands for C Compiler.

29
00:01:30,580 --> 00:01:35,600
Next I run the nm command,
which stands for names or name list.

30
00:01:35,610 --> 00:01:41,120
I use it on the object file,
to see the symbols located in it.

31
00:01:41,120 --> 00:01:45,120
nm outputs a list of addresses,
the type of each symbol,

32
00:01:45,120 --> 00:01:47,600
and then the symbol name.

33
00:01:47,600 --> 00:01:52,160
nm uses capital letters
to denote the types of global elements.

34
00:01:52,160 --> 00:01:56,080
As we can see,
the first symbol of hello.o

35
00:01:56,080 --> 00:01:59,660
is a globally undefined symbol: fprintf.

36
00:01:59,660 --> 00:02:02,540
Then there is a global symbol named global,

37
00:02:02,540 --> 00:02:04,880
which is of type capital "C".

38
00:02:04,880 --> 00:02:07,280
"C" stands for common area.

39
00:02:07,280 --> 00:02:10,780
It's the way global variables are defined in C.

40
00:02:10,780 --> 00:02:13,740
The name is a remnant
from old FORTRAN programs;

41
00:02:13,740 --> 00:02:16,300
a computer language
developed in the 1950s

42
00:02:16,300 --> 00:02:19,620
and still in use for scientific computing.

43
00:02:19,620 --> 00:02:21,560
After global there is local,

44
00:02:21,560 --> 00:02:25,840
a symbol that corresponds
to an uninitialized local function.

45
00:02:25,840 --> 00:02:30,120
Its type lowercase "b"
stands for block starting with symbol.

46
00:02:30,120 --> 00:02:31,280
In this way,

47
00:02:31,280 --> 00:02:35,620
local identifies a block
that is initialized to plain zeroes.

48
00:02:35,620 --> 00:02:40,600
The following lowercase "t" symbol
stands for text, which denotes code.

49
00:02:40,600 --> 00:02:44,040
It refers to the local function localfun.

50
00:02:44,040 --> 00:02:47,080
Furthermore,
we have another capital "T" symbol

51
00:02:47,080 --> 00:02:49,160
for the global function main.

52
00:02:49,160 --> 00:02:55,900
The last symbol here
is another undefined one: stdout.

53
00:02:56,700 --> 00:03:03,440
We can also use the nm command
on some executable files.

54
00:03:03,440 --> 00:03:07,500
I compile again hello.c,
but this time to an executable program

55
00:03:07,500 --> 00:03:09,660
instead of an object file.

56
00:03:09,660 --> 00:03:12,400
Running nm on the executable program hello,

57
00:03:12,400 --> 00:03:14,680
we get many more symbols now.

58
00:03:14,680 --> 00:03:16,560
That's because the program I wrote,

59
00:03:16,560 --> 00:03:20,780
has been compiled
and linked against the C library.

60
00:03:20,780 --> 00:03:21,800
For example,

61
00:03:21,800 --> 00:03:26,320
we see a capital "B" symbol,
block starting symbol bss_start,

62
00:03:26,320 --> 00:03:30,880
for the beginning address
of all C uninitialized variables.

63
00:03:30,880 --> 00:03:34,740
Executable programs
do not need their symbols to run.

64
00:03:34,740 --> 00:03:36,460
With the command called strip,

65
00:03:36,460 --> 00:03:39,920
we can remove all symbols
from an executable program.

66
00:03:39,920 --> 00:03:43,300
In this way,
I strip hello from its symbols.

67
00:03:43,300 --> 00:03:46,640
To verify that,
I run nm again on hello,

68
00:03:46,640 --> 00:03:50,080
and we get a warning
that hello contains no symbols.

69
00:03:50,080 --> 00:03:53,540
Production systems are often shipped
with their symbols stripped,

70
00:03:53,540 --> 00:03:57,480
in order to save disk space,
or prevent reverse engineering.

71
00:03:57,480 --> 00:04:00,280
For instance,
if I run nm on /bin/ls,

72
00:04:00,280 --> 00:04:05,560
we see that
there are no symbols available.

73
00:04:05,560 --> 00:04:09,300
As a complete example,
let's now use the nm command

74
00:04:09,300 --> 00:04:15,680
to find which source code files
contain the most global variables.

75
00:04:15,680 --> 00:04:18,040
I build this command step by step

76
00:04:18,040 --> 00:04:21,260
on a directory containing
all the compiled object files

77
00:04:21,260 --> 00:04:23,640
of the Apache web server.

78
00:04:23,640 --> 00:04:27,200
As a first step,
I run nm with the -o option,

79
00:04:27,200 --> 00:04:32,020
which means to precede each symbol 
with the filename where it occurs.

80
00:04:32,020 --> 00:04:36,380
I add as arguments to nm,
the output of the find command.

81
00:04:36,380 --> 00:04:39,340
find, searches in the current directory hierarchy

82
00:04:39,340 --> 00:04:42,000
for files ending with the suffix ".o";

83
00:04:42,000 --> 00:04:44,320
these are the object files.

84
00:04:44,320 --> 00:04:47,260
Then, for each symbol
found in the object files

85
00:04:47,260 --> 00:04:51,620
nm outputs the filename,
the symbol type and its name.

86
00:04:51,620 --> 00:04:55,760
This was the first step in finding
the file with the most global variables.

87
00:04:55,760 --> 00:04:59,820
As a second step,
I filter the output of nm

88
00:04:59,820 --> 00:05:03,500
to only list
the uninitialized global data definitions.

89
00:05:03,500 --> 00:05:07,820
To do that I pipe
the output of nm to grep capital "D",

90
00:05:07,820 --> 00:05:11,440
which represents
the uninitialized global variables.

91
00:05:11,440 --> 00:05:14,320
Indeed, we now see
symbols such as dir_module,

92
00:05:14,320 --> 00:05:17,040
action_module and auth_module.

93
00:05:17,040 --> 00:05:20,360
As a third step,
I run again my pipeline,

94
00:05:20,360 --> 00:05:23,540
but now I want to isolate the filenames.

95
00:05:23,540 --> 00:05:26,200
I use the cut command with the -d option,

96
00:05:26,200 --> 00:05:30,420
to specify as the field separator
the colon character.

97
00:05:30,420 --> 00:05:32,580
The filename is the first field,

98
00:05:32,580 --> 00:05:36,320
so I also specify that
with the -f option.

99
00:05:36,320 --> 00:05:39,180
As a result,
we get a list of filenames:

100
00:05:39,180 --> 00:05:42,280
mod_dir,
mod_actions, and so on.

101
00:05:42,280 --> 00:05:45,720
The final step
involves sorting the filenames

102
00:05:45,720 --> 00:05:48,580
by the number
of their global variables.

103
00:05:48,580 --> 00:05:52,060
I run again the pipeline
and pipe the output to sort,

104
00:05:52,060 --> 00:05:54,500
in order to group
the filenames together.

105
00:05:54,500 --> 00:05:56,680
Then I run uniq -c,

106
00:05:56,680 --> 00:05:59,840
to count the occurrences
of each distinct filename.

107
00:05:59,840 --> 00:06:00,800
Finally,

108
00:06:00,800 --> 00:06:04,460
I sort the filenames
in reverse numerical order.

109
00:06:04,460 --> 00:06:08,160
We see that http_main
contains the most global variables,

110
00:06:08,160 --> 00:06:11,060
37, quite a small number.

111
00:06:11,060 --> 00:06:17,020
The following files contain
either one or two variables each.

112
00:06:17,740 --> 00:06:21,280
On Unix and on Windows
most executable files are linked

113
00:06:21,280 --> 00:06:23,420
with dynamically linked libraries,

114
00:06:23,420 --> 00:06:26,260
also known as shared libraries.

115
00:06:26,260 --> 00:06:27,460
This allows the libraries

116
00:06:27,460 --> 00:06:30,740
to change independently
of the programs that are linked to,

117
00:06:30,740 --> 00:06:33,640
and also to save
some disk space.

118
00:06:33,640 --> 00:06:37,600
Let's see how we can find information
about which libraries are linked

119
00:06:37,600 --> 00:06:41,720
to which programs.

120
00:06:44,240 --> 00:06:47,260
I run the ldd command on /bin/ls

121
00:06:47,260 --> 00:06:50,300
to list all dynamically linked libraries.

122
00:06:50,300 --> 00:06:52,380
As an output for each library,

123
00:06:52,380 --> 00:06:55,800
we get the corresponding
system file it resolves to.

124
00:06:55,800 --> 00:06:58,540
To find the most used dynamic libraries,

125
00:06:58,540 --> 00:07:03,340
I first run ldd on all executable files
in the bin directory.

126
00:07:03,340 --> 00:07:05,820
Then, I pipe the output to sed,

127
00:07:05,820 --> 00:07:09,600
to get rid of the left hand side libraries
of the above list.

128
00:07:09,600 --> 00:07:13,140
I only want to keep
the system files the libraries resolve to;

129
00:07:13,140 --> 00:07:17,100
these are the right-hand side
parts of the list.

130
00:07:17,100 --> 00:07:19,040
Similar to the previous example,

131
00:07:19,040 --> 00:07:21,740
I run sort,
to group the libraries together.

132
00:07:21,740 --> 00:07:25,860
Then, I pipe the output to uniq -c
to count their occurrences,

133
00:07:25,860 --> 00:07:29,040
and sort the libraries
in reverse numerical order.

134
00:07:29,040 --> 00:07:32,400
With head,
we see the top ten used libraries.

135
00:07:32,410 --> 00:07:35,080
Three libraries share the first place.

136
00:07:35,080 --> 00:07:39,060
The Linux virtual dynamic
shared object library, vDSO,

137
00:07:39,060 --> 00:07:40,860
the C standard library,

138
00:07:40,860 --> 00:07:43,020
and the dynamic linker library.

139
00:07:43,020 --> 00:07:45,460
These are used by all programs.

140
00:07:45,460 --> 00:07:52,140
The following libraries
are used fewer times as we can see.

141
00:07:52,800 --> 00:07:56,320
In the Java ecosystem
we can use the javap command

142
00:07:56,320 --> 00:08:00,560
to look into Java class files.

143
00:08:00,560 --> 00:08:04,900
Running javap on a jar file
and specifying the file to a class

144
00:08:04,900 --> 00:08:07,340
we get the way this class is defined.

145
00:08:07,340 --> 00:08:10,500
The -c and -p arguments of javap

146
00:08:10,500 --> 00:08:14,800
are used to disassemble the file
and list the class's private members.

147
00:08:14,800 --> 00:08:15,940
As we can see,

148
00:08:15,940 --> 00:08:20,340
the output of javap is much more detailed
than the output of nm.

149
00:08:20,340 --> 00:08:22,000
That's because by default,

150
00:08:22,000 --> 00:08:29,000
Java class files contain
more information than compiled C programs.

151
00:08:29,000 --> 00:08:31,800
On Windows we can use
the dumpbin command

152
00:08:31,800 --> 00:08:36,320
in order to examine native code files.

153
00:08:37,240 --> 00:08:40,700
As an example,
let's change to the Windows directory

154
00:08:40,700 --> 00:08:44,620
and use dumpbin to see
what the Notepad program imports.

155
00:08:44,620 --> 00:08:47,600
I run dumpbin on the file notepad.exe,

156
00:08:47,600 --> 00:08:50,640
specifying as an argument "/imports".

157
00:08:50,640 --> 00:08:51,560
On Windows,

158
00:08:51,560 --> 00:08:55,320
arguments are specified
with a slash rather than a minus.

159
00:08:55,320 --> 00:08:58,280
We see that Notepad imports various functions,

160
00:08:58,280 --> 00:09:02,440
such as RegistrySetValue
and RegistryQueryValue.

161
00:09:02,440 --> 00:09:05,180
Furthermore, we can use the dumpbin command

162
00:09:05,180 --> 00:09:07,640
to see what a DLL exports.

163
00:09:07,640 --> 00:09:12,860
I change to the System32 directory
where many DLLs are stored.

164
00:09:12,860 --> 00:09:16,560
There, I run dumpbin on wsock32.dll,

165
00:09:16,560 --> 00:09:19,140
the Windows Socket implementation.

166
00:09:19,140 --> 00:09:21,920
By specifying the "/exports" argument

167
00:09:21,920 --> 00:09:24,100
we see that it exports various functions,

168
00:09:24,100 --> 00:09:29,360
such as Accept and EnumProtocols.

169
00:09:29,980 --> 00:09:33,480
This concludes our unit
on fetching data from compiled code.

170
00:09:33,480 --> 00:09:35,300
Stay with us!