cparameters.h

强化学习算法（R-Learning）难得的珍贵资料

// Copyright (C) 2003
// Gerhard Neumann ([e-mail])

//                
// This file is part of RL Toolbox.
// http://www.igi.tugraz.at/ril_toolbox
//
// All rights reserved.
// 
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions
// are met:
// 1. Redistributions of source code must retain the above copyright
//    notice, this list of conditions and the following disclaimer.
// 2. Redistributions in binary form must reproduce the above copyright
//    notice, this list of conditions and the following disclaimer in the
//    documentation and/or other materials provided with the distribution.
// 3. The name of the author may not be used to endorse or promote products
//    derived from this software without specific prior written permission.
// 
// THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
// IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
// OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
// IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
// INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
// NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
// THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


#ifndef C_PARAMETERS__H
#define C_PARAMETERS__H

#include <time.h>
#include <stdio.h>


#include "ril_debug.h"

#include <map>
#include <list>
#include <string>
#include <iostream>
#include <utility>

#define LINEAR 1
#define SQUARE 2
#define LOG	3
#define FRACT 4
#define FRACTSQUARE 5
#define FRACTLOG 6

using namespace std;

class CAdaptiveParameterCalculator;


/// This class represents a parameterset
/**
In the RL toolbox many objects, like learning algorithms have different parameters like the learning rate or the 
discount factor. The toolbox supports a general parameter handling for nearly all parameters of all the different classes. All these parameters can be set by a unified interface, which is supported by the class CParameters.
Every parameter in the toolbox is represented by its parameter name as a string and its parameter value. 
The toolbox provides methods to:
- set a Parameter value: In order to set a paramter just call the method setParameter(string parameterName, rlt_real value), with the name of the parameter and the value as argument. If you specify a parameter that doesn't exist you will get a warning message on the standard output, 
- get a Parameter value: Retrieve the value of a parameter with getParameter(string paramName). If you specify a name that doesn't exist you will get an assertation, so be careful when typing the names (or copy them).
- add a Parameter: A parameter can be added to the parameterset by the function addParameter, where you define the parameters name and its default value. If the parameter has already be added to the parameterset nothing happens (the default value won't be set too). Before a paramter is added, his value can't be set or retrieved.
Parameters are usually rlt_real values, but there are cases where integer or bool values are needed. If a parameter represents a integer value, it is rounded each time it is used. For boolean values, the >= 0.5 operator is used on the parameter value ( so 0.0 is false and 1.0 is true).
If you want to know more about the parameters of an object please consult the class reference of the specific object. The parameters set of an object can also be printed to standard output by calling the method saveParameters(stdout), which is a good way to see which parameters an specific object has.
The name and the value are stored in the parameters map of the CParameter object.
The class also provides methods for saving and loading parameter sets. You have the possibility to read/write the parameterset from/to a FILE stream, or a stream object. There is also the possibility to save the parameterset in xml style (without header).
For more details on parameter handling see the class reference of CParameterObject or the Manual.
*/

class CParameters
{
protected:
	std::map<string, rlt_real> *parameters;

public:
	/// Create new parameter set
	CParameters();

	/// Create new parameter set and copy the given reference
	CParameters(CParameters &copy);

	virtual ~CParameters();

	
	void loadParameters(FILE *stream);
	void saveParameters(FILE *stream);

	void loadParametersStream(istream *stream);
	void saveParametersStream(ostream *stream);

	/// save in XML-style
	/**
	 This is just pseudo-xml, so ne formatting rules or header is written.                                                                  
	*/
	void saveParametersXML(FILE *stream);
	/// load from XML-style stream
	void loadParametersXML(FILE *stream);

	/// Add the paramter with the given name and set value as default
	/**
	If the parameter is already in the parameter set, the default value is not set!
	*/
	virtual void addParameter(string name, rlt_real value);

	/// Add all paramters of the given parameter set to the current parameter set
	virtual void addParameters(CParameters *parameters);

	/// Remove parameter from set
	virtual void removeParameter(string name);

	/// returns the parameter value
	/**
	Attention: throws an assertation if the parameter is unknown!!!
	*/
	virtual rlt_real getParameter(string name);
	
	/// Set the parameter to the given value
	/**
	Prints a warning if the parameter is unknown
	*/
	virtual void setParameter(string name, rlt_real value);
	
	/// sets all parameters which are in both parameter set to the values of the given action set
	virtual void setParameters(CParameters *parameters);

	/// get parameter value from its index
	rlt_real getParameterFromIndex(unsigned int index);
	/// get parameter name from its index
	string getParameterName(unsigned int index);
	/// set parameter value with its index
	void setParameterWithIndex(unsigned int index, rlt_real value);

	/// get the index of the parameter
	int getParameterIndex(string name);

	int getNumParameters();

	/// returns true if all parameters are the same
	virtual bool operator == (CParameters &parameters);
	/// compares 2 parameter sets (needed for the parameter maps)
	bool operator < (CParameters &paramters);
};

/// Super class of all Objects mantaining parameters
/**
This class provides full parameter handling support for all its subclasses. Additionally to the functionality of the CParameters class, this class supports adding parameters of other CParameterObject's to the parameter set. The difference to normal to adding normal CParameter objects is, that if the parameter of the current object changes, all parameter objects which have been added by addParameters will get informed about the changed parameter. So if several objects have the same parameter you only have to set it once. Therefore the class contains a parameter object list to inform the other parameter objects
In the toolbox all parameter objects which are data elements of another parameter object get added to the parameter object itself, so all parameters of the data elements are parameters of the new class too. So for example the TD-Learner class contains the parameter "Lambda", even though this parameter initially belonged to the etrace object of the learner. If we change the parameter "Lambda" of the TD-Learner, it will also change the parameter "Lambda" for its etrace object. 
If 2 or more data elements have the same parameter, they can only have the same parameter value, because all of the parameter objects get informed about a parameter change. If this isn't desired, you can specify a parameter name prefix, when adding the parameter object to your new class. This Prefix is used to distinguish between the same parameter names of the parameter objects. Per default no prefix is used.
An additional functionality of parameter objects are adaptive paramter. For each parameter you can specify an adaptive parameter calculator, which calculates the parameter value each time it is retrieved. Now, each time the parameter's value is requested by "getParameter" the calculated value of the adaptive parameter calculator is returned instead of the constant rlt_real value of the parameter map. This is useful for example for adapting the learning rate or the exploration of a policy. The parameter's value can depend on any other value like the number of steps or episodes or even the current average reward. (see CAdaptiveParameterCalculator). Be aware that the adaptive parameter calculator is always set only for current object in the parameter object hierarchy. So if you set an adaptive parameter for the Parameter "Lambda" in a TD-Learner object, it won't affect the etraces, where the paremeter initially belong. So you have to set the adaptive parameter for the etrace object directly.
For performance reasons the parameter object subclasses have the possibility to not use the parameter set everytime they want to retrieve the parameters value, therefore they can store the parameters in rlt_real values, and each time a parameter value changes, they get informed by the function onParametersChanged. So this function has to be overwritten to update the rlt_real values if this is needed.
*/

class CParameterObject : public CParameters
{
protected:
	/// informs all parameter objects from the list and calls onParametersChanged
	void parametersChanged();
	

	typedef std::pair<CParameterObject *, string> paramPair;
	
	std::map<string, CAdaptiveParameterCalculator *> *adaptiveParameters;

	std::list<paramPair> *parameterObjects;
public:
	CParameterObject();
	virtual ~CParameterObject();

	/// Interface for faster parameter handling (see description of the class)
	virtual void onParametersChanged() {};

	/// sets the parameter and calls parametersChanged
	virtual void setParameter(string name, rlt_real value);
	/// sets the parameters and calls parametersChanged
	virtual void setParameters(CParameters *parameters);

	/// Add all parameters of the given parameter object to the current object, also add the given parameter object to the parameter object list
	/**