target_notes

iscsi源代码 UNH的progect 有initiator端和target端的源码

NOTES:	Notes on the target implementation

scsi_target_process_thread() - one for the SCSI MidLevel
    prints as "target_thread"
    big loop
	1. blocks on down_interruptible(&target_data.target_sem)
	2. goes through the target_data.msgq_start list looking at each
	   message.  Removes each list item in turn under control of
	   target_data.msg_lock spinlock.
	3. For each item, removes it under control of target_data.msg_lock
	   spinlock and then does switch(msg->message)
		ABORT_TASK		??
		LUN_RESET		scsi_release()
		TARGET_RESET		scsi_release()
		<others>		error
	4. goes through the target_data.cmd_queue_start list looking at
	   each command.
	5. For each item, does big if (cmd_curr->state ==
		ST_NEW_CMND		scsi_allocate_request()
					handle_cmd()
		ST_PENDING		hand_to_front_end()
		ST_TO_PROCESS		handle_cmd()
		ST_PROCESSED		read()
		ST_DONE			hand_to_front_end()
		ST_DEQUEUE		<remove from target_data.cmd_queue>
					under control of target_data.add_delete
					spinlock
					scsi_release_request()
		<others>		ignore

iscsi_server_thread() - one per TCP listening post
    big loop (no locks obtained or held)
	1. allocates a new socket
	2. blocks on accept()
	3. clean_bad_session()
	4. create and initialize new struct iscsi_conn for new connection
	5. create iscsi_tx_thread and iscsi_rx_thread for new connection

iscsi_tx_thread() - one per connection
    prints as "iscsi_tx_0"
    big loop (no locks obtained or held)
	1. blocks on down_interruptible(&conn->tx_sem)
	2. goes through session->cmnd_list looking at each struct iscsi_cmnd
	   on it for this conn  NOTE: list is NOT locked while doing this!
	3. does switch(cmnd->state) for each command belonging to this conn
		ISCSI_DISCOVERY		handle_discovery_rsp()
		ISCSI_LOGOUT		handle_logout_rsp()
		ISCSI_PING		handle_nopin()
		ISCSI_DONE		handle_iscsi_done()
					set by iscsi_xmit_response()
					which is called from midlevel
		ISCSI_MGT_FN_DONE	handle_iscsi_mgt_fn_done()
		ISCSI_BUFFER_RDY	iscsi_tx_r2t()
		ISCSI_DEQUEUE		iscsi_dequeue()
		ISCSI_QUEUE_CMND_RDY	send_unsolicited_data()
		<others>		<do nothing>

iscsi_rx_thread() - one per connection
    big loop (no locks obtained or held)
	1. calls iscsi_rx_data() to read a header
	2. if Header Digests in use, does crc on header,
	   then calls iscsi_rx_data() to read crc
	   then checks
	3. does switch(opcode) to various handle_xxx() routines
		LOGIN_CMND	handle_login()
		TEXT_CMND	handle_discovery()
		SCSI_CMND	handle_cmnd()
		SCSI_DATA_OUT	handle_data()
		TASK_MGMT_CMND	handle_task_mgt_command()
		LOGOUT_CMND	handle_logout()
		NOP_OUT		handle_nopout()
		SNACK		handle_snack()
		<others>	iscsi_tx_rjt()

Semaphores:
	devdata->session_sem		created UNLOCKED in iscsi_detect()
					mutex around devdata->session_list
	devdata->server_sem		created LOCKED in iscsi_detect()
					wait by ??? in bring_up_portal() after
					    starting up iscsi_server_thread
					wait by ??? in bring_down_portals()
					    after sending ISCSI_SHUTDOWN_SIGNAL
					    to iscsi_server_thread
					wakeup by iscsi_server_thread() after it
					    has started up ok
					wakeup by iscsi_server_thread() when it
					    exits
	session->cmnd_sem		mutex around the list session->cmnd_list
					and around all calls to scsi_rx_data()
					and around all calls to rx_cmnd()
					and around all calls to scsi_target_done
					and around all calls to check_cmd_sn()
					and update to session->cmnd_id
					and update to session->exp_cmd_sn
					and updates to session->max_cmd_sn
	session->conn_sem		mutex around the list session->conn_list
	cmnd->unsolicited_data_sem	created LOCKED in handle_cmnd()
					wait by rx thread when WRITE is received
					in order to give midlevel chance to set
					up the receive buffers
					wakeup by midlevel when it calls back to
					rdy_to_xfer which does it if immediate
					or unsolicited data is present
	conn->tx_sem			created LOCKED in iscsi_server_thread
					wait by tx thread at top of big loop
					wakeup 
	conn->kill_rx_sem		created LOCKED in iscsi_server_thread
					wait in iscsi_release_connection() after
					    sending ISCSI_SHUTDOWN_SIGNAL to rx
					    thread
					wakeup when rx thread exits
	conn->kill_tx_sem		created LOCKED in iscsi_server_thread
					wait in iscsi_release_connection() after
					    sending ISCSI_SHUTDOWN_SIGNAL to tx
					    thread
					wakeup when tx thread exits

In the iscsi layer, each command is represented by a struct iscsi_cmnd.
This struct is kept in session->cmnd_list during processing
    This list is protected by session->cmnd_sem
    Values in cmnd->state for commands in this list
	ISCSI_QUEUE_CMND	set in handle_cmnd()
				when new command is first received out of order
				means command is out-of-order by CmdSN
	ISCSI_QUEUE_CMND_RDY	set in iscsi_rdy_to_xfer() (called by midlevel)
				when prior state was ISCSI_QUEUE_CMND and
				when cmd->data_length > 0 and midlevel has set
				up buffers to accept input for WRITE command
	ISCSI_NEW_CMND		set in handle_cmnd()
				when new command is first received in order
				means command is in-order by CmdSN
	ISCSI_BUFFER_RDY	set in iscsi_rdy_to_xfer (called by midlevel)
				when cmd->data_length > 0 and midlevel has set
				up buffers to accept input for WRITE command
	ISCSI_ALL_R2TS_SENT	set in iscsi_tx_r2t (called by tx thread) when
				this command is a SCSI WRITE and ALL R2Ts have
				been sent on its behalf (ITT = command's ITT)
	ISCSI_DATA_IN		set in handle_cmnd() and send_unsolicited_data()
				means have finished reading the full amount of
				data to be sent in a WRITE command
	ISCSI_DONE		set in iscsi_rdy_to_xfer (called by midlevel)
				when cmd->data_length goes to 0
	ISCSI_SENT		means text rsp, logout rsp, nopin, scsi rsp,
				or a DataIn with S=1 have been sent to initiator
				set in handle_iscsi_done() and other places
				checked in (too) many places
	ISCSI_DEQUEUE		set in handle_cmnd() and other places when
				state is ISCSI_SENT
				means command's StatSN has been acked by
				initiator's ExpStatSN, tx thread needs to call
				midlevel's scsi_target_done() before freeing it
	ISCSI_MGT_FN_DONE	set in iscsi_task_mgt_fn_done (called by
				midlevel)
				means this command is a TMF and midlevel has
				finished with it

Flow of a READ_10 command

1. iscsi_rx_thread() reads 48-byte header into local buffer, reads and checks
   4-byte header digest (if needed), identifies opcode == ISCSI_INIT_SCSI_CMND
   and passes conn and header to handle_cmnd().

2. handle_cmnd() converts byte order in header fields, checks length, kmalloc's
   a struct iscsi_cmnd and initializes it from header, sets cmnd->state =
   ISCSI_NEW_CMND, and checks command order:
   a. If out of range, silently ignore it and return 0.
   b. If in range but out of order, sets cmnd->state = ISCSI_QUEUE_CMND, sets
      order = 1, and locks session->cmnd_sem.
   c. If in range and in order, locks session->cmnd_sem, calls rx_cmnd() and
      store result in cmnd->cmnd, and then calls check_queued_cmnd().
   Then finishes initializing new cmnd and then adds it to end of
   cmnd->session->cmnd_list, changing cmnd->state of other commands on that
   list to ISCSI_DEQUEUE if their state was ISCSI_SENT and their StatSN is
   acknowledged by new command's ExpStatSN.
   Then unlocks session->cmnd_sem.
   Then if order == 1 call save_unsolicited_data()
   else if immediate data present in command ...
   else if unsolicited data present in command ...

3. rx_cmnd in Midlevel kmallocs new struct Target_Scsi_Cmnd, initializes it
   (with copy of cdb and lun from pdu header), sets state = ST_NEW_CMND,
   spinlocks target_data.add_delete, sets id = ++targt_data.command_id,
   adds new struct to end of target_data.cmd_queue list, unlocks spinlock,
   does up on target_data.target_sem (which awakens scsi_target_process_thread),
   and returns pointer to new struct.

4. scsi_target_process_thread in Midlevel finds struct Target_Scsi_Cmnd on
   target_data.cmd_queue list, finds state = ST_NEW_CMND, kmallocs the req
   field in this command, zeros the new request structure, copies the cmd data
   (i.e., the cdb) into the req, and calls handle_cmd().

5. handle_cmd in Midlevel does switch on cdb opcode:
   READ_6, READ_10: calls get_allocation_length() to get number of bytes to_read
   from cdb, calls get_space() to allocate buffer for that number of bytes,
   set cmnd->state = ST_DONE.

6. on return from handle_cmd() (back to scsi_target_process_thread), detects
   state == ST_DONE and calls hand_to_front_end()

7. hand_to_front_end in Midlevel searches target_data.st_device_list for
   item with id = command->dev_id, finds that command->state == ST_DONE,
   calls curr_device->template->xmit_response, passing pointer to the command,
   sets command->state = ST_HANDED.

8. iscsi_xmit_response calls search_iscsi_cmnd to get back struct iscsi_cmnd
   *cmd to item whose cmnd field equals the cmnd passed into xmit_response,
   sets cmd->state = ISCSI_DONE, does up on cmd->cnn->tx_sem to wake up
   tx thread.

9. iscsi_tx_thread searches session->cmnd_list for cmnd with this conn, finds
   cmnd->state == ISCSI_DONE, calls handle_iscsi_done(cmnd, 0, 0).

10.handle_iscsi_done finds req->sr_data_direction == SCSI_DATA_READ and
   req->sr_result == DID_OK << 16, so it generates all DataIn PDU sequences
   data_length_left keeps track of bytes of data remaining to be sent
      (initializes this from req->sr_bufflen)
   while (data_length_left > 0) /* once around for each sequence/burst */
      data_send_length = max possible to send in this burst (sequence)
      last_seq = 1 if this is last burst needed by this command
      offset = offset into req->sr of first byte to be sent in this burst
      seq_offset = offset int req->sr of first byte to be sent in next PDU
      reduce data_length_left by data_send_length
      data_send = bytes sent so far in this burst (initially 0)
      while (data_send_length > 0) /* once around for each DataIn PDU */
         initialize local iscsi_hdr with ISCSI_TARG_SCSI_DATA_IN
         data_payload_length = max possible to send in next pdu
         hdr->flags == F_BIT if this is last pdu in this burst
         hdr->flags == S_BIT if this is last pdu in last burst
	 increase data_send by data_payload_length
         st_list = req->sr_buffer + cmnd->scatter_list_count
         niov = find_iovec_needed() + header  + hdr_crc + padding + data_crc
         iov = kmalloc niov's
         cmnd->scatter_list_count += fill_iov()
         pad_bytes = kmalloc padding
         call iscsi_tx_data()
         kfree(iov)
         kfree(pad_bytes)
         decrease data_send_length by data_payload_length
         increase offset by data_payload_length
   set cmnd->state = ISCSI_SENT

Target Transfer Tag at offset 20 in all reply PDUs
	the field name in the PDU is: target_xfer_tag
	the value of this field is set to ALL_ONES in:
			handle_nopin()
			handle_iscsi_done()
			handle_discovery_rsp()
		it is set to ++cmnd->session->cmnd_id in:
			handle_cmnd()
		it is copied from the iscsi_cmnd in:
			iscsi_tx_r2t()

    It is used in the following replies sent by target:
	ISCSI_TARG_SCSI_DATA_IN	 10.7 DataIn		handle_iscsi_done()
	ISCSI_TARG_R2T		 10.8 R2T		iscsi_tx_r2t()
	ISCSI_TARG_TEXT_RSP	 10.11 Text Response	handle_discovery_rsp()
	ISCSI_TARG_NOP_IN	 10.19 NopIn		handle_nopin()

    It is NOT used in the following replies sent by target:
	ISCSI_TARG_SCSI_RSP	 10.4 SCSI Response
	ISCSI_TARG_TASK_MGMT_RSP 10.6 Task Managment Function Response
	ISCSI_TARG_ASYNC_MSG	 10.9 Asynchronous Message
	ISCSI_TARG_LOGIN_RSP	 10.13 Login Response
	ISCSI_TARG_LOGOUT_RSP	 10.15 Logout Response
	ISCSI_TARG_RJT		 10.17 Reject

    It is used in the following requests sent by initiator:
	ISCSI_INIT_SCSI_DATA_OUT 10.7 DataOut		handle_data()
	ISCSI_INIT_TEXT_CMND	 10.10 Text Request	handle_discovery()
	ISCSI_INIT_SNACK	 10.16 SNACK Request	handle_snack()
	ISCSI_INIT_NOP_OUT	 10.18 NopOut		handle_nopout()
	
    It is NOT used in the following requests sent by initiator:
	ISCSI_INIT_SCSI_CMND	 10.3 SCSI Command	handle_cmnd()
	ISCSI_INIT_TASK_MGMT_CMND10.5 Task Management	handle_task_mgt_command
	ISCSI_INIT_LOGIN_CMND	 10.12 Login Request	handle_login()
	ISCSI_INIT_LOGOUT_CMND	 10.14 Logout Request	handle_logout()

during a WRITE command processing
cmnd->data_length	is total number of bytes expected from ini
cmnd->data_done		is total number of bytes read from initiator so far
			(which is also the offset when things are in order)
			this includes immediate, unsolicited, solicited bytes
			are done when cmnd->data_done >= cmnd->data_length
cmnd->next_burst_len	is number of (solicited) bytes read for this R2T so far
			so it should never exceed MaxBurstLength
cmnd->first_burst_len	is total number of unsolicited bytes read so far
			so it should never exceed FirstBurstLength
cmnd->r2t_data		is number of bytes that need to be solicited but that
			have not yet been asked for by any R2Ts
			initialized in handle_cmnd() when cmnd first set up
			tested in handle_data() to see if need to send more R2Ts
			tested and decremented in iscsi_tx_r2t()as R2Ts are sent
			done sending R2Ts when this is no longer positive