debugging390.txt

嵌入式系统设计与实例开发实验教材二源码 多线程应用程序设计 串行端口程序设计 AD接口实验 CAN总线通信实验 GPS通信实验 Linux内核移植与编译实验 IC卡读写实验 SD驱动使

killall -HUP inetd
or ps aux | grep inetd
get inetd's process id
& kill -HUP inetd to restart it.

Important options
-----------------
-o is used to tell strace to output to a file in our case t1 in the root directory
-f is to follow children i.e.
e.g in our case above telnetd will start the login process & subsequently a shell like bash.
You will be able to tell which is which from the process ID's listed on the left hand side
of the strace output.
-p<pid> will tell strace to attach to a running process, yup this can be done provided
 it isn't being traced or debugged already & you have enough privileges,
the reason 2 processes cannot trace or debug the same program is that strace
becomes the parent process of the one being debugged & processes ( unlike people )
can have only one parent.


However the file /t1 will get big quite quickly
to test it telnet 127.0.0.1

now look at what files in.telnetd execve'd
413   execve("/usr/sbin/in.telnetd", ["/usr/sbin/in.telnetd", "-h"], [/* 17 vars */]) = 0
414   execve("/bin/login", ["/bin/login", "-h", "localhost", "-p"], [/* 2 vars */]) = 0 

Whey it worked!.


Other hints:
------------
If the program is not very interactive ( i.e. not much keyboard input )
& is crashing in one architecture but not in another you can do 
an strace of both programs under as identical a scenario as you can
on both architectures outputting to a file then.
do a diff of the two traces using the diff program
i.e.
diff output1 output2
& maybe you'll be able to see where the call paths differed, this
is possibly near the cause of the crash. 

More info
---------
Look at man pages for strace & the various syscalls
e.g. man strace, man alarm, man socket.


Performance Debugging
=====================
gcc is capible of compiling in profiling code just add the -p option
to the CFLAGS, this obviously affects program size & performance.
This can be used by the gprof gnu profiling tool or the
gcov the gnu code coverage tool ( code coverage is a means of testing
code quality by checking if all the code in an executable in exercised by
a tester ).


Using top to find out where processes are sleeping in the kernel
----------------------------------------------------------------
To do this copy the System.map from the root directory where
the linux kernel was built to the /boot directory on your 
linux machine.
Start top
Now type fU<return>
You should see a new field called WCHAN which
tells you where each process is sleeping here is a typical output.
 
 6:59pm  up 41 min,  1 user,  load average: 0.00, 0.00, 0.00
28 processes: 27 sleeping, 1 running, 0 zombie, 0 stopped
CPU states:  0.0% user,  0.1% system,  0.0% nice, 99.8% idle
Mem:   254900K av,   45976K used,  208924K free,       0K shrd,   28636K buff
Swap:       0K av,       0K used,       0K free                    8620K cached

  PID USER     PRI  NI  SIZE  RSS SHARE WCHAN     STAT  LIB %CPU %MEM   TIME COMMAND
  750 root      12   0   848  848   700 do_select S       0  0.1  0.3   0:00 in.telnetd
  767 root      16   0  1140 1140   964           R       0  0.1  0.4   0:00 top
    1 root       8   0   212  212   180 do_select S       0  0.0  0.0   0:00 init
    2 root       9   0     0    0     0 down_inte SW      0  0.0  0.0   0:00 kmcheck

The time command
----------------
Another related command is the time command which gives you an indication
of where a process is spending the majority of its time.
e.g.
time ping -c 5 nc
outputs
real	0m4.054s
user	0m0.010s
sys	0m0.010s

Debugging under VM
==================

Notes
-----
Addresses & values in the VM debugger are always hex never decimal
Address ranges are of the format <HexValue1>-<HexValue2> or <HexValue1>.<HexValue2> 
e.g. The address range  0x2000 to 0x3000 can be described described as
2000-3000 or 2000.1000

The VM Debugger is case insensitive.

VM's strengths are usually other debuggers weaknesses you can get at any resource
no matter how sensitive e.g. memory managment resources,change address translation
in the PSW. For kernel hacking you will reap dividends if you get good at it.

The VM Debugger displays operators but not operands, probably because some
of it was written when memory was expensive & the programmer was probably proud that
it fitted into 2k of memory & the programmers & didn't want to shock hardcore VM'ers by
changing the interface :-), also the debugger displays useful information on the same line & 
the author of the code probably felt that it was a good idea not to go over 
the 80 columns on the screen. 

As some of you are probably in a panic now this isn't as unintuitive as it may seem
as the 390 instructions are easy to decode mentally & you can make a good guess at a lot 
of them as all the operands are nibble ( half byte aligned ) & if you have an objdump listing
also it is quite easy to follow, if you don't have an objdump listing keep a copy of
the s/390 Reference Summary & look at between pages 2 & 7 or alternatively the
s/390 principles of operation.
e.g. even I can guess that 
0001AFF8' LR    180F        CC 0
is a ( load register ) lr r0,r15 

Also it is very easy to tell the length of a 390 instruction from the 2 most significant
bits in the instruction ( not that this info is really useful except if you are trying to
make sense of a hexdump of code ).
Here is a table
Bits                    Instruction Length
------------------------------------------
00                          2 Bytes
01                          4 Bytes
10                          4 Bytes
11                          6 Bytes




The debugger also displays other useful info on the same line such as the
addresses being operated on destination addresses of branches & condition codes.
e.g.  
00019736' AHI   A7DAFF0E    CC 1
000198BA' BRC   A7840004 -> 000198C2'   CC 0
000198CE' STM   900EF068 >> 0FA95E78    CC 2



Useful VM debugger commands
---------------------------

I suppose I'd better mention this before I start
to list the current active traces do 
Q TR
there can be a maximum of 255 of these per set
( more about trace sets later ).
To stop traces issue a
TR END.
To delete a particular breakpoint issue
TR DEL <breakpoint number>

The PA1 key drops to CP mode so you can issue debugger commands,
Doing alt c (on my 3270 console at least ) clears the screen. 
hitting b <enter> comes back to the running operating system
from cp mode ( in our case linux ).
It is typically useful to add shortcuts to your profile.exec file
if you have one ( this is roughly equivalent to autoexec.bat in DOS ).
file here are a few from mine.
/* this gives me command history on issuing f12 */
set pf12 retrieve 
/* this continues */
set pf8 imm b
/* goes to trace set a */
set pf1 imm tr goto a
/* goes to trace set b */
set pf2 imm tr goto b
/* goes to trace set c */
set pf3 imm tr goto c



Instruction Tracing
-------------------
Setting a simple breakpoint
TR I PSWA <address>
To debug a particular function try
TR I R <function address range>
TR I on its own will single step.
TR I DATA <MNEMONIC> <OPTIONAL RANGE> will trace for particular mnemonics
e.g.
TR I DATA 4D R 0197BC.4000
will trace for BAS'es ( opcode 4D ) in the range 0197BC.4000
if you were inclined you could add traces for all branch instructions &
suffix them with the run prefix so you would have a backtrace on screen 
when a program crashes.
TR BR <INTO OR FROM> will trace branches into or out of an address.
e.g.
TR BR INTO 0 is often quite useful if a program is getting awkward & deciding
to branch to 0 & crashing as this will stop at the address before in jumps to 0.
TR I R <address range> RUN cmd d g
single steps a range of addresses but stays running &
displays the gprs on each step.



Displaying & modifying Registers
--------------------------------
D G will display all the gprs
Adding a extra G to all the commands is neccessary to access the full 64 bit 
content in VM on z/Architecture obviously this isn't required for access registers
as these are still 32 bit.
e.g. DGG instead of DG 
D X will display all the control registers
D AR will display all the access registers
D AR4-7 will display access registers 4 to 7
CPU ALL D G will display the GRPS of all CPUS in the configuration
D PSW will display the current PSW
st PSW 2000 will put the value 2000 into the PSW &
cause crash your machine.
D PREFIX displays the prefix offset


Displaying Memory
-----------------
To display memory mapped using the current PSW's mapping try
D <range>
To make VM display a message each time it hits a particular address & continue try
D I<range> will disassemble/display a range of instructions.
ST addr 32 bit word will store a 32 bit aligned address
D T<range> will display the EBCDIC in an address ( if you are that way inclined )
D R<range> will display real addresses ( without DAT ) but with prefixing.
There are other complex options to display if you need to get at say home space
but are in primary space the easiest thing to do is to temporarily
modify the PSW to the other addressing mode, display the stuff & then
restore it.


 
Hints
-----
If you want to issue a debugger command without halting your virtual machine with the
PA1 key try prefixing the command with #CP e.g.
#cp tr i pswa 2000
also suffixing most debugger commands with RUN will cause them not
to stop just display the mnemonic at the current instruction on the console.
If you have several breakpoints you want to put into your program &
you get fed up of cross referencing with System.map
you can do the following trick for several symbols.
grep do_signal System.map 
which emits the following among other things
0001f4e0 T do_signal 
now you can do

TR I PSWA 0001f4e0 cmd msg * do_signal
This sends a message to your own console each time do_signal is entered.
( As an aside I wrote a perl script once which automatically generated a REXX
script with breakpoints on every kernel procedure, this isn't a good idea
because there are thousands of these routines & VM can only set 255 breakpoints
at a time so you nearly had to spend as long pruning the file down as you would 
entering the msg's by hand ),however, the trick might be useful for a single object file.
On linux'es 3270 emulator x3270 there is a very useful option under the file ment
Save Screens In File this is very good of keeping a copy of traces. 

From CMS help <command name> will give you online help on a particular command. 
e.g. 
HELP DISPLAY

Also CP has a file called profile.exec which automatically gets called
on startup of CMS ( like autoexec.bat ), keeping on a DOS analogy session
CP has a feature similar to doskey, it may be useful for you to
use profile.exec to define some keystrokes. 
e.g.
SET PF9 IMM B
This does a single step in VM on pressing F8. 
SET PF10  ^
This sets up the ^ key.
which can be used for ^c (ctrl-c),^z (ctrl-z) which can't be typed directly into some 3270 consoles.
SET PF11 ^-
This types the starting keystrokes for a sysrq see SysRq below.
SET PF12 RETRIEVE
This retrieves command history on pressing F12.


Sometimes in VM the display is set up to scroll automatically this
can be very annoying if there are messages you wish to look at
to stop this do
TERM MORE 255 255
This will nearly stop automatic screen updates, however it will
cause a denial of service if lots of messages go to the 3270 console,
so it would be foolish to use this as the default on a production machine.
 

Tracing particular processes
----------------------------
The kernels text segment is intentionally at an address in memory that it will
very seldom collide with text segments of user programs ( thanks Martin ),
this simplifies debugging the kernel.
However it is quite common for user processes to have addresses which collide
this can make debugging a particular process under VM painful under normal
circumstances as the process may change when doing a 
TR I R <address range>.
Thankfully after reading VM's online help I figured out how to debug
particular processes in 31 bit mode, however, according to the current 
VM online help documentation the method described below uses 
TR STO or STD which don't currently work on  z/Series while in 
64-bit mode.

Your first problem is to find the STD ( segment table designation )
of the program you wish to debug.

There are several ways you can do this here are a few
1) objdump --syms <program to be debugged> | grep main
To get the address of main in the program.
tr i pswa <address of main>
Start the program, if VM drops to CP on what looks like the entry
point of the main function this is most likely the process you wish to debug.
Now do a D X13 or D XG13 on z/Architecture.
On 31 bit the STD is bits 1-19 ( the STO segment table origin ) 
& 25-31 ( the STL segment table length ) of CR13.
now type
TR I R STD <CR13's value> 0.7fffffff
e.g.
TR I R STD 8F32E1FF 0.7fffffff
Another very useful variation is
TR STORE INTO STD <CR13's value> <address range>
for finding out when a particular variable changes.

An alternative way of finding the STD of a currently running process 
is to do the following, ( this method is more complex but
could be quite convient if you aren't updating the kernel much &
so your kernel structures will stay constant for a reasonable period of
time ).