target_read

iscsi源代码 UNH的progect 有initiator端和target端的源码

Walking thru a READ command on the UNH target.

State
-----
There are 2 components to the state of a command at any time, written as:
	[iscsi-state, target-state]
where iscsi-state names start with ISCSI_xxx and represents the state seen
	by the rx_thread and tx_thread in iscsi_target.c,
and target-state names start with ST_xxx and represents the state seen
	by the target_thread in scsi_target.c.


State Transitions
-----------------
State at the end of each step in the diagrams that follow.
0.	[<null>, <null>]
1.	[ISCSI_NEW_CMND, ST_NEW_CMND]
2.	[ISCSI_NEW_CMND, ST_PROCESSING]
3.	[ISCSI_NEW_CMND, ST_DONE]
4.	[ISCSI_DONE, ST_HANDED]
5.	[ISCSI_SENT, ST_HANDED]
6.	[ISCSI_DEUEUE, ST_HANDED]
7.	[<null>, ST_DEQUEUE]
8.	[<null>, <null>]


Thread Interactions
-------------------
	rx_thread	tx_thread	target_thread	midlevel
	|		|		|		|
	1 ---------------------------->
					2 ------------>
					  <------------	3
			  <------------ 4
			5
	6 ------------>
			7 ------------>
					8


Overview
--------
0.		[<null>, <null>]

1. rx_thread	gets new READ pdu from initiator, allocates iscsi command for it
		and adds this to iscsi queue, then calls-back rx_cmnd() in
		target to allocate target command, add it to target queue,
		and then wake up target_thread.
		[ISCSI_NEW_CMND, ST_NEW_CMND]

2. target_thread allocates scsi buffer space and calls-back scsi_do_req() in
		midlevel to wake up midlevel
		[ISCSI_NEW_CMND, ST_PROCESSING]

3. midlevel	actually reads data from device into buffers, then calls-back
		te_cmnd_processed() in target to wake up target_thread
		[ISCSI_NEW_CMND, ST_DONE]

4. target_thread calls-back xmit_response() in iscsi to wake up tx_thread
		[ISCSI_DONE, ST_HANDED]

5. tx_thread	creates sequences of DataIn pdus from filled data buffers and
		sends them to initiator, then sends SCSI Response pdu (unless
		phase collapse set S bit on last DataIn pdu sent).
		[ISCSI_SENT, ST_HANDED]

6. rx_thread	gets any pdu from initiator that acks the StatSN of the SCSI
		Response pdu, then wakes up tx_thread
		[ISCSI_DEQUEUE, ST_HANDED]

7. tx_thread	removes command from iscsi queue, calls-back scsi_target_done()
		in target to wake up target_thread, and frees iscsi command
		[<null>, ST_DEQUEUE]

8. target_thread frees scsi buffer space, removes command from target queue,
		then frees the target command
		[<null>, <null>]

Details
-------
1. rx_thread in iscsi_target.c
	-> main loop in iscsi_rx_thread()
		read SCSI Command Header (opcode 0x01)
		read and check header crc if header digest is on
		dispatch on opcode to case ISCSI_INIT_SCSI_CMND
		which calls handle_cmnd()
	-> handle_cmnd()
		convert big-endian to host pdu fields IN PLACE
		call get_new_cmnd() to allocate clean struct iscsi_cmnd *cmnd
		cmnd->state = ISCSI_NEW_CMND
		fill in fields in cmnd from pdu
		call check_cmd_sn() to check CmdSN of pdu for in-order cmnd
		if cmnd is out-of-order,
			cmnd->state = ISCSI_QUEUE_CMND
			call ack_sent_cmnds() to add to session->cmnd_list
		else
			call ack_sent_cmnds() to add to session->cmnd_list
			call-back target's rx_cmnd() to set up buffers
	---------> rx_cmnd() in scsi_target.c
			allocate Target_Scsi_Cmnd *command
			fill in fields in command, including cdb
			command->state = ST_NEW_CMND
			assign command->id
			add command to end of target_data.cmd_queue
			up on target_data.target_sem
		<--------
	<--------
	At this point:	cmnd (state=ISCSI_NEW_CMND) is in session->cmnd_list,
			command (state=ST_NEW_CMND) is in target_data.cmd_queue
			target must act next

2. target_thread in scsi_target.c
	-> main loop in scsi_target_process_thread()
		wake up after blocking on target_data.target_sem
		look at each command in target_data.cmd_queue
			when command->state == ST_NEW_CMND
			#ifdef DISKIO
				map [target_id, lun] to this_device
				call-back mid-level's scsi_allocate_request() to
					set up Scsi_Request *command->req
			#else
				kmalloc Scsi_Request *command->req and clear it
			#endif
			copy cdb into *command->req
			call handle_cmd()
	-----------------> handle_cmd()
			   #ifdef MEMORYIO
			   #else ifdef DISKIO
				#ifdef TRUST_CDB
					set command->req->sr_data_direction
						 = SCSI_DATA_READ
					call get_space() to allocate
						scatter-gather list
					command->state = ST_PROCESSING
					call-back scsi_do_req() in mid-level
				#else
					dispatch on cdb opcode
					case READ_10:
						command->req->sr_data_direction
							 = SCSI_DATA_READ
						call get_allocation_length() to
							get allocation length
							from cdb
						call get_space() to allocate
							scatter-gather list
						command->state = ST_PROCESSING
						call-back scsi_do_req() in
							mid-level
	<----------------------------------------
				#endif
			   #else ifdef GENERICIO
			   #else ifdef FILEIO

	At this point:	cmnd (state=ISCSI_NEW_CMND) is in session->cmnd_list,
			command (state=ST_PROCESSING)is in target_data.cmd_queue
			mid-level must act next

3. scsi subsystem mid-level
	do the READ request
		fill the buffers with data from disk
		fill in the req's sr_sense_buffer with sense data
		call-back te_cmnd_processed()
	-> te_cmnd_processed() in scsi_target.c
		search target_data.cmd_queue to find command with this req
		if found
			command->state = ST_DONE
			up target_data.target_sem to wake up target thread
		else
			call scsi_release_request() to throw it away
	<----------------

	At this point:	cmnd (state=ISCSI_NEW_CMND) is in session->cmnd_list,
			command (state=ST_DONE) is in target_data.cmd_queue
			target must act next

4. target_thread in scsi_target.c
	-> main loop in scsi_target_process_thread()
		wake up after blocking on target_data.target_sem
		look at each command in target_data.cmd_queue
			when command->state == ST_DONE
				call hand_to_front_end()
		----------------> hand_to_front_end()
					search target_data.st_device_list for
					command's device
					#ifdef GENERICIO
					#endif
					command->state = ST_HANDED
					call-back xmit_response()
		------------------------> xmit_response() in iscsi_target.c
						call search_iscsi_cmnd() to find
							matching cmnd
						bump session's exp_cmd_sn
						cmnd->state = ISCSI_DONE
						up on conn->tx_sem
				<----------------
	<-----------------------

	At this point:	cmnd (state=ISCSI_DONE) is in session->cmnd_list,
			command (state=ST_HANDED) is in target_data.cmd_queue
			tx_thread must act next

5. tx_thread in iscsi_target.c
	-> main loop in iscsi_tx_thread()
		wake up after blocking on conn->tx_sem
		look at each command in session->cmnd_list
			dispatch on state to case ISCSI_DONE
			which calls handle_iscsi_done()
			repeat loop

	-> handle_iscsi_done()
		when have both req->sr_data_direction == SCSI_DATA_READ and
		req->sr_result is DID_OK (i.e., mid-level finished ?)
			call send_read_data() to send all DataIn pdus to init
	----------------> send_read_data()
				call do_command_status() to find out SCSI status
	------------------------> do_command_status()
					data_length_left = size of req's buffer
					(this should be == cmnd->data_length)
					check sense data in req->sr_sense_buffer
						and set appropriate flags
					check for under/overflow and set flags
					return flags, data_length_left to send
				<-------- return to send_read_data()
				if not retransmitting,
					copy cmnd->data_sn to cmnd->prev_data_sn
				while data_length_left > 0
					send one sequence each time around loop
					seq_length = no. of bytes in sequence
					set LAST_SEQ_FLAG if last sequence
					data_length_left -= seq_length
					use Orders to set limits of this sequenc
					while seq_length > 0
						send 1 DataIn pdu on each loop
						fill in header for DataIn pdu
						data_payload_length = data size
						set F bit on last pdu in sequenc
						set A bit if enabled
						if phase collapsing and no sense
							set S bit on last pdu in
								command
						fill in rest of DataIn pdu headr
						set up iovector
						seq_length -=data_payload_length
						deal with padding, digests, etc.
						call iscsi_tx_data() to send
							DataIn pdu
					end loop for sending 1 DataIn pdu
					adjust sequence limits
				end loop for sending 1 sequence of DataIn pdus
			<-------- return to send_read_data()
			if this was a retransmission
			cmnd->state = ISCSI_SENT
			else if no phase_collapse then
				call send_iscsi_response()
				-> send_iscsi_response()
					fill in fields in SCSI Response pdu
					when have both req->sr_data_direction
							== SCSI_DATA_READ and
							req->sr_result is DID_OK
						call do_command_status() to find
							out SCSI status
					fill in under/over flow
					if SEND_SENSE_FLAG was set by
							do_command_status()
						set up sense data in data
							segment of response pdu
					cmnd->retransmit_flg = 0
					cmnd->state = ISCSI_SENT
					call send_hdr_plus_1_data() to send
						SCSI Response pdu
					up on conn->tx_sem
				<-------- return to send_read_data()
			else
				cmnd->state = ISCSI_SENT
			call check_queued_cmnd() to deal with queued
				out-of-order cmnds
			up on conn->tx_sem
		<-------- return from handle_iscsi_done() to main tx_thread loop

	At this point:	cmnd (state=ISCSI_SENT) is in session->cmnd_list,
			command (state=ST_HANDED) is in target_data.cmd_queue
			rx_thread must act next when initiator confirms StatSN

6. rx_thread in iscsi_target.c
	-> main loop in iscsi_rx_thread()
		read next pdu from initiator
		read and check header crc if header digest is on
		dispatch on opcode to case 
		which calls a specific handle_xxx()
		all of which call ack_sent_cmnds() to use their ExpStatSN field
	---------> ack_sent_cmnds()
			loops through each command in session->cmnd_list
				if command's state == ISCSI_SENT and
					command's StatSN < new ExpStatSN
				then	this command's StatSN has been acked by
						initiator
					set state = ISCSI_DEQUEUE
					up on conn->tx_sem
				if caller wants to queue this command
					add it to end of session->cmnd_list
				up conn->tx_sem

	At this point:	cmnd (state=ISCSI_DEQUEUE) is in session->cmnd_list,
			command (state=ST_HANDED) is in target_data.cmd_queue
			tx_thread must act next

7. tx_thread in iscsi_target.c
	-> main loop in iscsi_tx_thread()
		wake up after blocking on conn->tx_sem
		look at each command in session->cmnd_list
			dispatch on state to case ISCSI_DEQUEUE
			which removes command from list & calls iscsi_dequeue()
			then restarts loop

	-> iscsi_dequeue()
		if pointer to SCSI cmnd is not NULL
			call-back scsi_target_done() in scsi_target.c
	----------------> scsi_target_done()
				state = ST_DEQUEUE
				up target_data.target_sem
		if this was connection's text_in_progess command, mark it freed
		free up structures pointed to by command
		free up command structure itself

	At this point:	cmnd is free and is NOT in session->cmnd_list,
			command (state=ST_DEQUEUE) is in target_data.cmd_queue
			target thread must act next

8. target_thread in scsi_target.c
	-> main loop in scsi_target_process_thread()
		wake up after blocking on target_data.target_sem
		look at each command in target_data.cmd_queue
			when command->state == ST_DEQUEUE
				free up pages in command's req scatterlist
				call scsi_release_request() to free commands req
				remove command from target_data.cmd_queue
				free command structure

	At this point:	cmnd is free and is NOT in session->cmnd_list,
			command is free and is NOT in target_data.cmd_queue
			no more action needed on target side

ERRORS so far:
done	1. Daren is correct, those lines should be removed.
done	2. test for last_seq == 1 to set phase collapse should only be done
	   when send_sense is 0! Otherwise, must send scsi response pdu with
	   the sense data!
done	3. Why are send_iscsi_response() and handle_iscsi_done() mostly
	   duplicates???
	4. Should we do cmnd->state = ISCSI_MTG_FN_DONE at the end of
	   do_task_mgt(), or should mid-level do this when it is really
	   finished? (Minti)
ok	5. Look at latest SCSI specs to check Sense Data Format (page 240)
done	6. in send_iscsi_response(), err_code no longer gets set to
	   anything useful!  Was set before in do_command_status() when
	   sense_buffer[0] was 0x70, and this is now being lost.
	   Need to look at SEND_SENSE_FLAG instead!
	   But as it is now, sense data will NEVER be sent!