caction.h

强化学习算法（R-Learning）难得的珍贵资料

// An action which can long for several steps.
/** An action with a duration different than one has to be treated with SemiMarkov-Learning rules in all
the learning algorithms, since otherwise the result can be far from optimal. Many Learning-Algorithm in the this package support
Semi-Markov Learning updates.
This action class maintains its own action data object, a CMultiStepActionData object. This data object stores the duration it has needed by now and the finished flag. The duration and the finished flag get normally set by an HierarchicalController, but it is also possible to set the duration for example in the environment model, if you have a primitive action which takes longer than other primitive actions. In that case your model-specific action has to be derivated from CPrimitiveAction and CMultiStepAction.
\par
The class also contains the method isFinished(CStateCollection *state). This method is used (normally by a hierarchical controller) to determine
wether the action has finished or not. The controller sets the finished flag according to isFinished, so other Listeners only have to
look at this flag. This method must be implemented by all (non-abstract) sub-classes.
*/

class CMultiStepAction :  public CAction
{
protected:
	CMultiStepActionData *multiStepData;

	CMultiStepAction(CMultiStepActionData *multiStepData);
public:
	CMultiStepAction();


	/**
	This method is normally used by a hierarchical controller to determine
	wether the action has finished or not. The finsished method may depend only on the current state transition, so you get the old state and the new state as parameters. The controller sets the finished flag according to isFinished, so other Listeners only have to
	look at this flag. This method must be implemented by all (non-abstract) sub-classes.
	*/
	virtual bool isFinished(CStateCollection *oldState, CStateCollection *newState) = 0;

	virtual CActionData *getNewActionData();

	virtual CMultiStepActionData *getMultiStepActionData() {return multiStepData;};

	/// returns the duration of the action (member: duration)
	virtual int getDuration() {return multiStepData->duration;};
};

// Represents a primitive Action
/**
The only kind of actions which can be added to the Agent and passed to the EnvironmentModel as action to execute.
For a specific learning problem you have to derivate your ModelActions from this class and add some specific attributes
to the action (for example force...).
The type PRIMITIVEACTION is added to the type field of the action.
@see CPrimitiveActionStateChange
*/

class CPrimitiveAction : public CAction
{
protected:
	CPrimitiveAction(CMultiStepActionData *actionData);
public:
	CPrimitiveAction();
	~CPrimitiveAction();
	
	virtual int getDuration() {return 1;};
};

/// This abstract class represents extended actions like behaviors or hierarchical SMDPs.
/** The CExtendedAction class can represent behaviors and other actions which are composed of
other primitive, or even extended actions. Since an extended action usually consists of a composition of several other, "more primitive" actions, the extended action 
has also an duration, so its derivated from CMultiStepAction. Like an agent controller you can retrieve an other action from the
extended action to execute (with getNextHierarchyLevel(...)). The action returned by this method can be a primitive action or aswell an extended action with lower hierarchy. 
Gathering the next action until a primitiv action occurs is done by the class CHierarchicalController, 
meanwhile the Hierarchical Stack is also created. The Hierachic Controller also sets the nextHierarchyLevel Pointer according to getNextHierarchyLevel(...).
<p>
When using extended actions you have the possibility that all intermediate steps which occured during the execution of the
extended action get send to the Listeners by the class CSemiMarkovDecisionProcess. For sending the intermediate steps to a agent listener the function "intermediateStep" is used instead of "nextStep", the different function is needed because intermediate steps has to be treated differently in some cases (ETraces). To get the intermediate steps of a executed behavior you take all states 
occured during the execution of the action. To create a S-A-S tuple you take one state of that list of states for the first state in the tuple (soo its the "current" state), set the duration of the extended action correctly and 
for the second state of the tuple you take the state in which the action has finished. This is only usefull for behaviors 
which finishing condition only depends on the actual state. With this method your are able to provide more training examples.
If you don't wan
*/
class CExtendedAction : public CMultiStepAction
{
protected:
	CExtendedAction(CMultiStepActionData *actionData);


public:
	/// Pointer to the action executed by the extended Action.
	CAction *nextHierarchyLevel;

	CExtendedAction();

/// Virtual function for determining the next action in the next hierarchie level.
	virtual CAction* getNextHierarchyLevel(CStateCollection *state, CActionDataSet *actionDataSet = NULL) = 0;
/// Constructs a hierarchical ActionStack, with the extended action itself as root.
	void getHierarchicalStack(CHierarchicalStack *actionStack);
/// Flag for sending the intermediate Steps of this action.
	bool sendIntermediateSteps;
};




/// class maintaining all the actions available for a certain object (normally a CActionObject like a controller) 
/** This class maintains a list of all actions which should be useable for another object.
It only saves the pointers of the action Objects, so you can't save for example the duration of an action, if
you change the duration later (which is normally the case), for that case use CActionList. The pointer of the action is also used as a kind of "Id". According 
to the pointer the function getIndex(...) returns the index of the action, so the other objects can determine which 
action was chosen.
/par 
CActionSet also provides a function for gett韓g all available actions in the current State from the action set.
@see CActionList
*/
class CActionSet : public std::list<CAction *>
{

public:
	CActionSet();
	~CActionSet();

/// returns the index of the action.
	int getIndex(CAction *action);

/// returns the index of the index th action.
	CAction *get(unsigned int index);

/// add an action to the set.
	void add(CAction *action);

/// add all actions from an action set to the action set
	void add(CActionSet *actions);

/// get all available actions in the current State from the action set.
	void getAvailableActions(CActionSet *availableActions, CStateCollection *state);

//	returns wether the given action is member of the actionset
	bool isMember(CAction *action);
};

/// Base Class for all Classes which have to maintain an action set.
/** It just has a pointer to the actionset an some functions to get information
about the actionset */
class CActionObject  
{
protected:
	CActionSet *actions;
	bool ownActionSet;
public:
	CActionObject(CActionSet *actions, bool createNew = false);
	~CActionObject();

	CActionSet *getActions();
	unsigned int getNumActions();

};


/// class for mantaining the action data objects of all actions of an action set
/** This class is needed mainly for controllers, since they are not allowed to change the data of the action itself, they maintain a local action dataset, and modify the data of the data set. 
*/
class CActionDataSet : public CActionObject
{
protected:
	std::map<CAction *, CActionData *> *actionDatas;
public:
	CActionDataSet(CActionSet *actions);
	~CActionDataSet();

	CActionData *getActionData(CAction *action);
	void setActionData(CAction *action, CActionData *data);

	void addActionData(CAction *action);
	void removeActionData(CAction *action);
};

/// class for logging a sequence of actions
/** This class saves the ActionData Objects and the index of the actions in separate lists. 
If you add an action, the index is stored in the index list (so the action has to be member of the 
actionlist's actionset. If the specific action returns a valid CActionData Object, this is also stored in the actionDatas map.
/par
If you want to get an action with a specified number in the sequence, the action with the index specified by 
actionIndices[num] is returned, but before if there is an action data, it is set. 
*/
class CActionList : public CActionObject
{
protected:
	// vector for the action indices
	std::vector<int> *actionIndices;
	/** map for the actionDatas, the mapIndex is the actionNumber in the sequence, because not all action have an action data*/
	std::map<int, CActionData *> *actionDatas;
public:
	CActionList(CActionSet *actions);
	virtual ~CActionList();


	void addAction(CAction *action);
	CAction *getAction(unsigned int num, CActionDataSet *l_data);

	virtual void loadBIN(FILE *stream);
	virtual void saveBIN(FILE *stream);

	virtual void loadASCII(FILE *stream);
	virtual void saveASCII(FILE *stream);
	
	/// Returns the number of actions in the action list.
	unsigned int getSize();
	unsigned int getNumActions();
	

	void clear();
};



#endif