Performance Testers

Metric-based Test Framework

The metric-based test framework can be used to create test metrics on all sorts of data gathered by the software at run time. The base implementation offers a variety of services to save and load data, parse LLSD results, compare current performance with a base and produce test reports.

The interface is implemented in llcommon/llmetricperformancetester.h. Two abstract classes are available:

• LLMetricPerformanceTesterBasic
• LLMetricPerformanceTesterWithSession

LLMetricPerformanceTesterBasic

The abstract class LLMetricPerformanceTesterBasic defines the general metric-based test framework.

This class can be directly inherited from for simple data gathering and provides predefined methods to save, load and compare results of performance sessions.

Below is the detailed doc for this class.

/**
 * @class LLMetricPerformanceTesterBasic
 * @brief Performance Metric Base Class
 */
class LL_COMMON_API LLMetricPerformanceTesterBasic
{
public:
    /**
     * @brief Creates a basic tester instance.
     * @param[in] name - Unique string identifying this tester instance.
     */
	LLMetricPerformanceTesterBasic(std::string name);
	virtual ~LLMetricPerformanceTesterBasic();

    /**
     * @return Returns true if the instance has been added to the tester map.
     * Need to be tested after creation of a tester instance so to know if the tester is correctly handled.
     * A tester might not be added to the map if another tester with the same name already exists.
     */
    BOOL isValid() const { return mValidInstance; }

    /**
     * @brief Write a set of test results to the log LLSD.
     */
	void outputTestResults() ;
    
    /**
     * @brief Compare the test results.
     * By default, compares the test results against the baseline one by one, item by item, 
     * in the increasing order of the LLSD record counter, starting from the first one.
     */
	virtual void analyzePerformance(std::ofstream* os, LLSD* base, LLSD* current) ;
    
    /**
     * @return Returns the number of the test metrics in this tester instance.
     */
	S32 getNumberOfMetrics() const { return mMetricStrings.size() ;}
    /**
     * @return Returns the metric name at index
     * @param[in] index - Index on the list of metrics managed by this tester instance.
     */
	std::string getMetricName(S32 index) const { return mMetricStrings[index] ;}
    
protected:
    /**
     * @return Returns the name of this tester instance.
     */
	std::string getTesterName() const { return mName ;}
    
    /**
     * @brief Insert a new metric to be managed by this tester instance.
     * @param[in] str - Unique string identifying the new metric.
     */
	void addMetric(std::string str) ;

    /**
     * @brief Compare test results, provided in 2 flavors: compare integers and compare floats.
     * @param[out] os - Formatted output string holding the compared values.
     * @param[in] metric_string - Name of the metric.
     * @param[in] v_base - Base value of the metric.
     * @param[in] v_current - Current value of the metric.
     */
	virtual void compareTestResults(std::ofstream* os, std::string metric_string, S32 v_base, S32 v_current) ;
	virtual void compareTestResults(std::ofstream* os, std::string metric_string, F32 v_base, F32 v_current) ;
    
    /**
     * @brief Reset internal record count. Count starts with 1.
     */
	void resetCurrentCount() { mCount = 1; }
    /**
     * @brief Increment internal record count.
     */
	void incrementCurrentCount() { mCount++; }
    /**
     * @return Returns the label to be used for the current count. It's "TesterName"-"Count".
     */
    std::string getCurrentLabelName() const { return llformat("%s-%d", mName.c_str(), mCount) ;}
    
    /**
     * @brief Write a test record to the LLSD. Implementers need to overload this method.
     * @param[out] sd - The LLSD record to store metric data into.
     */
	virtual void outputTestRecord(LLSD* sd) = 0 ;

private:
	void preOutputTestResults(LLSD* sd) ;
	void postOutputTestResults(LLSD* sd) ;

	std::string mName ;                         // Name of this tester instance
	S32 mCount ;                                // Current record count
    BOOL mValidInstance;                            // TRUE if the instance is managed by the map
	std::vector< std::string > mMetricStrings ; // Metrics strings

// Static members managing the collection of testers
public:	
    // Map of all the tester instances in use
	typedef std::map< std::string, LLMetricPerformanceTesterBasic* > name_tester_map_t;	
	static name_tester_map_t sTesterMap ;

    /**
     * @return Returns a pointer to the tester
     * @param[in] name - Name of the tester instance queried.
     */
	static LLMetricPerformanceTesterBasic* getTester(std::string name) ;
    /**
     * @return Returns TRUE if there's a tester defined, FALSE otherwise.
     */
	static BOOL hasMetricPerformanceTesters() { return !sTesterMap.empty() ;}
    /**
     * @brief Delete all testers and reset the tester map
     */
	static void cleanClass() ;

private:
    // Add a tester to the map. Returns false if adding fails.
	static BOOL addTester(LLMetricPerformanceTesterBasic* tester) ;    
};

LLMetricPerformanceTesterWithSession

The abstract class LLMetricPerformanceTesterWithSession is derived from the previous one and provides an additional session abstraction that allow the definition of ad-hoc comparison method for reporting.

This class should be used when data need to be collated and analyzed in specific ways.

Below is the detailed doc for this class.

/**
 * @class LLMetricPerformanceTesterWithSession
 * @brief Performance Metric Class with custom session 
 */
class LL_COMMON_API LLMetricPerformanceTesterWithSession : public LLMetricPerformanceTesterBasic
{
public:
    /**
     * @param[in] name - Unique string identifying this tester instance.
     */
	LLMetricPerformanceTesterWithSession(std::string name);
	virtual ~LLMetricPerformanceTesterWithSession();

    /**
     * @brief Compare the test results.
     * This will be loading the base and current sessions and compare them using the virtual 
     * abstract methods loadTestSession() and compareTestSessions()
     */
	virtual void analyzePerformance(std::ofstream* os, LLSD* base, LLSD* current) ;

protected:
    /**
     * @class LLMetricPerformanceTesterWithSession::LLTestSession
     * @brief Defines an interface for the two abstract virtual functions loadTestSession() and compareTestSessions()
     */
	class LLTestSession
        {
        public:
            virtual ~LLTestSession() ;
        };
    
    /**
     * @brief Convert an LLSD log into a test session.
     * @param[in] log - The LLSD record
     * @return Returns the record as a test session
     */
	virtual LLMetricPerformanceTesterWithSession::LLTestSession* loadTestSession(LLSD* log) = 0;
    
    /**
     * @brief Compare the base session and the target session. Assumes base and current sessions have been loaded.
     * @param[out] os - The comparison result as a standard stream
     */
	virtual void compareTestSessions(std::ofstream* os) = 0;
    
	LLTestSession* mBaseSessionp;
	LLTestSession* mCurrentSessionp;
};

Creating/Adding a Basic Test Metrics

First, you need to create a metric-based tester class derived from the LLMetricPerformanceTesterBasic class and that will hold your performance data. The key steps are as following.

• declare your own tester derived from LLMetricPerformanceTesterBasic
• in the constructor, declare all metrics you will use in this tester, the declaration order does not matter
• collect the test data in your own way. The usual way is to define an update() method that gets called and gather the relevant performance data.
• define the abstract virtual method outputTestRecord(LLSD* sd) to output your test data to the LLSD structure. Everything output to the LLSD in this function will be saved to the log file metric.slp in the log folder.
• the final test report contains the following columns: metric_string, baseline_value, target_value, target_value - baseline_value and 100 * target_value / baseline_value.

Below is a code example:

class YourOwnTester : public LLMetricPerformanceTesterBasic
{
public:
    YourOwnTester() ;
    ~YourOwnTester() ;

    // This will have to get called in code to update your perf data.
    // Note: you can create as many updateXx() variation as your perf system requires
    void update(const S32 d1, const F32 d2) ;

protected:
    // This is required. It tells the class how to pack the data in an LLSD stream
    /*virtual*/ void outputTestRecord(LLSD* sd) ;

private:
    // Define the relevant perf gathering variables. 
    // Note: the default compare method only supports S32 and F32 comparison. You need to overload the compare if you need to carry something else.
    S32 data1;
    F32 data2;
    ...
};

YourOwnTester::YourOwnTester() : LLMetricPerformanceTesterBasic("your-unique-tester-name-string")
{
    // Declare all the metrics used in the tester.
    addMetric("metric-string-1") ;
    addMetric("metric-string-2") ;
    ...

    // Your own initializations
    data1 = 0;
    data2 = 0.0f;
    ...
}

YourOwnTester::~YourOwnTester()
{
    // You likely need to invalidate the static pointer holding that test instance
    sYourTester = NULL;
}

void YourOwnTester::outputTestRecord(LLSd *sd)
{
    std::string currentLabel = getCurrentLabelName();
    //insert your own code to output test results to sd
    //format like this
    (*sd)[currentLabel]["metric-string-1"] = (LLSD::Integer)data1;
    (*sd)[currentLabel]["metric-string-2"] = (LLSD::Real)data2;
    ...
} 

void YourOwnTester::update(const S32 d1, const F32 d2)
{
    // Do something with the input data to update your perf data
    data1 += d1;
    data2 += d2;
    ...

    // *Important* You need to call outputTestResults() when some perf gathering condition is met
    // Otherwise your data might not be saved to the log ever.
    if (condition)
    {
        outputTestResults();
    }
}

You may check the class LLImageCompressionTester as an example.

How To Run Metric-based Automated Test

Performance metric gathering must be done in 2 main steps:

• Baseline performance data acquisition
• Target run and analysis

Baseline Performance Data Acquisition

During that step, simply run the viewer so to gather initial data. You'll use that run to create a metric_baseline.slp file that will be used later to compare new runs with this baseline. If you are for instance on the verge of doing modification to improve the perf of some part of the viewer, this is something you want to do before doing any modification to the code so you can later measure you much your modifications improved (or not) the performance measured.

Note that every time you add new performance metric, you need to re-create a baseline. It is therefore advisable to develop perf metric code in a non-modified repository and pull the perf metric code to the modified repository while developing new algorithms. It is actually not uncommon to get ideas of new perf data to gather while doing performance tuning development.

Steps:

• Make sure the log folder does not contain any previously gathered data, i.e. delete or backup the files metric.slp and metric_baseline.slp
• Launch the viewer from the command line with -logmetrics as an argument
• Perform the test session
• Quit the viewer: at that point, you should have a new metric.slp file in the log folder. Note that quitting might take some time as writing the metric.slp file is time consuming. Do not force quit the viewer during that stage.
• Rename the file metric.slp to metric_baseline.slp and leave it in the log folder

Target Performance Data Acquisition and Analysis

You can run the target run and analysis using the same baseline over and over. The basic data are saved in a metric.slp file and the data comparison is stored in a metric_report.csv file that can be open with Excel or Open Office.

Note that the code overwrites old metric.slp and metric_report.csv files so save your reports if you need to.

Steps:

• Make sure the log folder contains a metric_baseline.slp file. The perf analysis will crash on quit if that file is not present.
• Launch the viewer from the command line with -logmetrics -analyzeperformance as arguments
• Perform the test session
• Quit the viewer: you can find the test results in the file metric_report.csv located in your secondlife log file folder.

More

If one wants to gather data automatically, it's possible to use -autologin and -replaysession on the command line.