debugging390.txt

嵌入式系统设计与实例开发实验教材二源码 多线程应用程序设计 串行端口程序设计 AD接口实验 CAN总线通信实验 GPS通信实验 Linux内核移植与编译实验 IC卡读写实验 SD驱动使

              
                          Debugging on Linux for s/390 & z/Architecture
			               by
		Denis Joseph Barrow ([e-mail],[e-mail])
		Copyright (C) 2000-2001 IBM Deutschland Entwicklung GmbH, IBM Corporation
                              Best viewed with fixed width fonts 

Overview of Document:
=====================
This document is intended to give an good overview of how to debug 
Linux for s/390 & z/Architecture it isn't intended as a complete reference & not a
tutorial on the fundamentals of C & assembly, it dosen't go into
390 IO in any detail. It is intended to compliment the documents in the
reference section below & any other worthwhile references you get.

It is intended like the Enterprise Systems Architecture/390 Reference Summary
to be printed out & used as a quick cheat sheet self help style reference when
problems occur.

Contents
========
Register Set
Address Spaces on Intel Linux
Address Spaces on Linux for s/390 & z/Architecture
The Linux for s/390 & z/Architecture Kernel Task Structure
Register Usage & Stackframes on Linux for s/390 & z/Architecture
A sample program with comments
Compiling programs for debugging on Linux for s/390 & z/Architecture
Figuring out gcc compile errors
Debugging Tools
objdump
strace
Performance Debugging 
Debugging under VM
s/390 & z/Architecture IO Overview
Debugging IO on s/390 & z/Architecture under VM
GDB on s/390 & z/Architecture
Stack chaining in gdb by hand
Examining core dumps
ldd
Debugging modules
The proc file system
Starting points for debugging scripting languages etc.
SysRq
References
Special Thanks

Register Set
============
The current architectures have the following registers.
 
16  General propose registers, 32 bit on s/390 64 bit on z/Architecture, r0-r15 or gpr0-gpr15 used for arithmetic & addressing. 

16 Control registers, 32 bit on s/390 64 bit on z/Architecture, ( cr0-cr15 kernel usage only ) used for memory managment,
interrupt control,debugging control etc.

16 Access registers ( ar0-ar15 ) 32 bit on s/390 & z/Architecture
not used by normal programs but potentially could 
be used as temporary storage. Their main purpose is their 1 to 1
association with general purpose registers and are used in
the kernel for copying data between kernel & user address spaces.
Access register 0 ( & access register 1 on z/Architecture ( needs 64 bit 
pointer ) ) is currently used by the pthread library as a pointer to
the current running threads private area.

16 64 bit floating point registers (fp0-fp15 ) IEEE & HFP floating 
point format compliant on G5 upwards & a Floating point control reg (FPC) 
4  64 bit registers (fp0,fp2,fp4 & fp6) HFP only on older machines.
Note:
Linux (currently) always uses IEEE & emulates G5 IEEE format on older machines,
( provided the kernel is configured for this ).


The PSW is the most important register on the machine it
is 64 bit on s/390 & 128 bit on z/Architecture & serves the roles of 
a program counter (pc), condition code register,memory space designator.
In IBM standard notation I am counting bit 0 as the MSB.
It has several advantages over a normal program counter
in that you can change address translation & program counter 
in a single instruction. To change address translation,
e.g. switching address translation off requires that you
have a logical=physical mapping for the address you are
currently running at.

      Bit           Value
s/390 z/Architecture
0       0     Reserved ( must be 0 ) otherwise specification exception occurs.

1       1     Program Event Recording 1 PER enabled, 
	      PER is used to facilititate debugging e.g. single stepping.

2-4    2-4    Reserved ( must be 0 ). 

5       5     Dynamic address translation 1=DAT on.

6       6     Input/Output interrupt Mask

7       7     External interrupt Mask used primarily for interprocessor signalling & 
	      clock interupts.

8-11  8-11    PSW Key used for complex memory protection mechanism not used under linux

12      12    1 on s/390 0 on z/Architecture

13      13    Machine Check Mask 1=enable machine check interrupts

14      14    Wait State set this to 1 to stop the processor except for interrupts & give 
	      time to other LPARS used in CPU idle in the kernel to increase overall 
	      usage of processor resources.

15      15    Problem state ( if set to 1 certain instructions are disabled )
	      all linux user programs run with this bit 1 
	      ( useful info for debugging under VM ).

16-17 16-17   Address Space Control

	      00 Primary Space Mode when DAT on
	      The linux kernel currently runs in this mode, CR1 is affiliated with 
              this mode & points to the primary segment table origin etc.

	      01 Access register mode this mode is used in functions to 
	      copy data between kernel & user space.

	      10 Secondary space mode not used in linux however CR7 the
	      register affiliated with this mode is & this & normally
	      CR13=CR7 to allow us to copy data between kernel & user space.
	      We do this as follows:
	      We set ar2 to 0 to designate its
	      affiliated gpr ( gpr2 )to point to primary=kernel space.
	      We set ar4 to 1 to designate its
	      affiliated gpr ( gpr4 ) to point to secondary=home=user space
	      & then essentially do a memcopy(gpr2,gpr4,size) to
	      copy data between the address spaces, the reason we use home space for the
	      kernel & don't keep secondary space free is that code will not run in 
	      secondary space.

	      11 Home Space Mode all user programs run in this mode.
	      it is affiliated with CR13.

18-19 18-19   Condition codes (CC)

20    20      Fixed point overflow mask if 1=FPU exceptions for this event 
              occur ( normally 0 ) 

21    21      Decimal overflow mask if 1=FPU exceptions for this event occur 
              ( normally 0 )

22    22      Exponent underflow mask if 1=FPU exceptions for this event occur 
              ( normally 0 )

23    23      Significance Mask if 1=FPU exceptions for this event occur 
              ( normally 0 )

24-31 24-30   Reserved Must be 0.

      31      Extended Addressing Mode
      32      Basic Addressing Mode
              Used to set addressing mode
	      PSW 31   PSW 32
                0         0        24 bit
                0         1        31 bit
                1         1        64 bit

32             1=31 bit addressing mode 0=24 bit addressing mode (for backward 
               compatibility ), linux always runs with this bit set to 1

33-64          Instruction address.
      33-63    Reserved must be 0
      64-127   Address
               In 24 bits mode bits 64-103=0 bits 104-127 Address 
               In 31 bits mode bits 64-96=0 bits 97-127 Address
               Note: unlike 31 bit mode on s/390 bit 96 must be zero
	       when loading the address with LPSWE otherwise a 
               specification exception occurs, LPSW is fully backward
               compatible.
 	  
	  
Prefix Page(s)
--------------	  
This per cpu memory area is too intimately tied to the processor not to mention.
It exists between the real addresses 0-4096 on s/390 & 0-8192 z/Architecture & is exchanged 
with a 1 page on s/390 or 2 pages on z/Architecture in absolute storage by the set 
prefix instruction in linux'es startup. 
This page is mapped to a different prefix for each processor in an SMP configuration
( assuming the os designer is sane of course :-) ).
Bytes 0-512 ( 200 hex ) on s/390 & 0-512,4096-4544,4604-5119 currently on z/Architecture 
are used by the processor itself for holding such information as exception indications & 
entry points for exceptions.
Bytes after 0xc00 hex are used by linux for per processor globals on s/390 & z/Architecture 
( there is a gap on z/Architecure too currently between 0xc00 & 1000 which linux uses ).
The closest thing to this on traditional architectures is the interrupt
vector table. This is a good thing & does simplify some of the kernel coding
however it means that we now cannot catch stray NULL pointers in the
kernel without hard coded checks.



Address Spaces on Intel Linux
=============================

The traditional Intel Linux is approximately mapped as follows forgive
the ascii art.
0xFFFFFFFF 4GB Himem                        *****************
                                            *               *
                                            * Kernel Space  *
                                            *               *
                                            *****************          ****************
User Space Himem (typically 0xC0000000 3GB )*  User Stack   *          *              *
				            *****************          *              *
					    *  Shared Libs  *          * Next Process *          
                                            *****************          *     to       *  
					    *               *    <==   *     Run      *  <==
					    *  User Program *          *              *
					    *   Data BSS    *          *              *
                                            *	 Text       *          *              *
         			            *   Sections    *          *              *
0x00000000         			    *****************          ****************

Now it is easy to see that on Intel it is quite easy to recognise a kernel address 
as being one greater than user space himem ( in this case 0xC0000000).
& addresses of less than this are the ones in the current running program on this
processor ( if an smp box ).
If using the virtual machine ( VM ) as a debugger it is quite difficult to
know which user process is running as the address space you are looking at
could be from any process in the run queue.

The limitation of Intels addressing technique is that the linux
kernel uses a very simple real address to virtual addressing technique
of Real Address=Virtual Address-User Space Himem.
This means that on Intel the kernel linux can typically only address
Himem=0xFFFFFFFF-0xC0000000=1GB & this is all the RAM these machines
can typically use.
They can lower User Himem to 2GB or lower & thus be
able to use 2GB of RAM however this shrinks the maximum size
of User Space from 3GB to 2GB they have a no win limit of 4GB unless
they go to 64 Bit.


On 390 our limitations & strengths make us slightly different.
For backward compatibility ( because of the psw address hi bit which
indicates whether we are in 31 or 64 bit mode ) we are only allowed 
use 31 bits (2GB) of our 32 bit addresses. However, 
we use entirely separate address  spaces for the user & kernel.

This means we can support 2GB of non Extended RAM on s/390, & more
with the Extended memory managment swap device & 
currently 4TB of physical memory currently on z/Architecture.


Address Spaces on Linux for s/390 & z/Architecture
==================================================

Our addressing scheme is as follows


Himem 0x7fffffff 2GB on s/390    *****************          ****************
currently 0x3ffffffffff (2^42)-1 *  User Stack   *          *              *
on z/Architecture.		 *****************          *              *
		                 *  Shared Libs  *          *              *      
                                 *****************          *              *  
			         *               *          *    Kernel    *  
		                 *  User Program *          *              *
		                 *   Data BSS    *          *              *
                                 *    Text       *          *              *
            			 *   Sections    *          *              *
0x00000000                       *****************          ****************

This also means that we need to look at the PSW problem state bit
or the addressing mode to decide whether we are looking at
user or kernel space.

Virtual Addresses on s/390 & z/Architecture
===========================================

A virtual address on s/390 is made up of 3 parts
The SX ( segment index, roughly corresponding to the PGD & PMD in linux terminology ) 
being bits 1-11.
The PX ( page index, corresponding to the page table entry (pte) in linux terminology )
being bits 12-19. 
The remaining bits BX (the byte index are the offset in the page )
i.e. bits 20 to 31.

On z/Architecture in linux we currently make up an address from 4 parts.
The region index bits (RX) 0-32 we currently use bits 22-32
The segment index (SX) being bits 33-43
The page index (PX) being bits  44-51
The byte index (BX) being bits  52-63

Notes:
1) s/390 has no PMD so the PMD is really the PGD also.
A lot of this stuff is defined in pgtable.h.

2) Also seeing as s/390's page indexes are only 1k  in size 
(bits 12-19 x 4 bytes per pte ) we use 1 ( page 4k )
to make the best use of memory by updating 4 segment indices 
entries each time we mess with a PMD & use offsets 
0,1024,2048 & 3072 in this page as for our segment indexes.
On z/Architecture our page indexes are now 2k in size
( bits 12-19 x 8 bytes per pte ) we do a similar trick
but only mess with 2 segment indices each time we mess with
a PMD.

3) As z/Architecture supports upto a massive 5-level page table lookup we 
can only use 3 currently on Linux ( as this is all the generic kernel
currently supports ) however this may change in future
this allows us to access ( according to my sums )
4TB of virtual storage per process i.e.
4096*512(PTES)*1024(PMDS)*2048(PGD) = 4398046511104 bytes,
enough for another 2 or 3 of years I think :-).
to do this we use a region-third-table designation type in
our address space control registers.
 

The Linux for s/390 & z/Architecture Kernel Task Structure
==========================================================
Each process/thread under Linux for S390 has its own kernel task_struct
defined in linux/include/linux/sched.h
The S390 on initialisation & resuming of a process on a cpu sets
the __LC_KERNEL_STACK variable in the spare prefix area for this cpu
( which we use for per processor globals).

The kernel stack pointer is intimately tied with the task stucture for
each processor as follows.

                      s/390
            ************************
            *  1 page kernel stack *
	    *        ( 4K )        *
            ************************
            *   1 page task_struct *