lab.reports – Make reports

lab.reports.arithmetic_mean(values)[source]

Compute the arithmetic mean of a sequence of numbers.

>>> arithmetic_mean([20, 30, 70])
40.0
lab.reports.geometric_mean(values)[source]

Compute the geometric mean of a sequence of numbers.

>>> round(geometric_mean([2, 8]), 2)
4.0
class lab.reports.Attribute(name, absolute=False, min_wins=True, function=None, functions=None, scale=None, digits=2)[source]

A string subclass for attributes in reports.

Use this class if your custom attribute needs a non-default value for:

  • absolute: if False, only include tasks for which all task runs have values in a per-domain table (e.g. coverage is absolute, whereas expansions is not, because we can’t compare algorithms A and B for task X if B has no value for expansions).
  • min_wins: set to True if a smaller value for this attribute is better, to False if a higher value is better and to None if values can’t be compared. (E.g., min_wins is False for coverage, but it is True for expansions).
  • function: the function used to aggregate values of multiple runs for this attribute, for example, in domain reports. Defaults to sum().
  • functions: deprecated. Pass a single function instead.
  • scale: default scaling. Can be one of “linear”, “log” and “symlog”. If scale is None (default), the reports will choose the scaling.
  • digits: number of digits after the decimal point.

The downward package automatically uses appropriate settings for most attributes.

>>> avg_h = Attribute("avg_h", min_wins=False)
>>> abstraction_done = Attribute(
...     "abstraction_done", absolute=True, min_wins=False
... )
class lab.reports.Report(attributes=None, format='html', filter=None, **kwargs)[source]

Base class for all reports.

Inherit from this or a child class to implement a custom report.

Depending on the type of output you want to make, you will have to overwrite the write(), get_text() or get_markup() method.

attributes is the list of attributes you want to include in your report. If omitted, use all numerical attributes. Globbing characters * and ? are allowed. Example:

>>> report = Report(attributes=["coverage", "translator_*"])

When a report is made, both the available and the selected attributes are printed on the commandline.

format can be one of e.g. html, tex, wiki (MediaWiki), doku (DokuWiki), pmw (PmWiki), moin (MoinMoin) and txt (Plain text). Subclasses may allow additional formats.

If given, filter must be a function or a list of functions that are passed a dictionary of a run’s attribute keys and values. Filters must return True, False or a new dictionary. Depending on the returned value, the run is included or excluded from the report, or replaced by the new dictionary, respectively.

Filters for properties can be given in shorter form without defining a function. To include only runs where attribute foo has value v, use filter_foo=v. To include only runs where attribute foo has value v1, v2 or v3, use filter_foo=[v1, v2, v3].

Filters are applied sequentially, i.e., the first filter is applied to all runs before the second filter is executed. Filters given as filter_* kwargs are applied after all filters passed via the filter kwarg.

Examples:

Include only the “cost” attribute in a LaTeX report:

>>> report = Report(attributes=["cost"], format="tex")

Only include successful runs in the report:

>>> report = Report(filter_coverage=1)

Only include runs in the report where the initial h value is at most 100:

>>> def low_init_h(run):
...     return run["initial_h_value"] <= 100
...
>>> report = Report(filter=low_init_h)

Only include runs from “blocks” and “barman” with a timeout:

>>> report = Report(filter_domain=["blocks", "barman"], filter_search_timeout=1)

Add a new attribute:

>>> def add_expansions_per_time(run):
...     expansions = run.get("expansions")
...     time = run.get("search_time")
...     if expansions is not None and time:
...         run["expansions_per_time"] = expansions / time
...     return run
...
>>> report = Report(
...     attributes=["expansions_per_time"], filter=[add_expansions_per_time]
... )

Rename, filter and sort algorithms:

>>> def rename_algorithms(run):
...     name = run["algorithm"]
...     paper_names = {"lama11": "LAMA 2011", "fdss_sat1": "FDSS 1"}
...     run["algorithm"] = paper_names[name]
...     return run
...
>>> # We want LAMA 2011 to be the leftmost column.
>>> # filter_* filters are evaluated last, so we use the updated
>>> # algorithm names here.
>>> algorithms = ["LAMA 2011", "FDSS 1"]
>>> report = Report(filter=rename_algorithms, filter_algorithm=algorithms)
__call__(eval_dir, outfile)[source]

Make the report.

This method is called automatically when the report step is executed. It loads the data and calls write().

eval_dir must be a path to an evaluation directory containing a properties file.

The report will be written to outfile.

get_markup()[source]

Return txt2tags markup for the report.

get_text()[source]

Return text (e.g., HTML, LaTeX, etc.) for the report.

By default this method calls get_markup() and converts the markup to the desired output format.

write()[source]

Write the report files.

By default this method calls get_text() and writes the obtained text to outfile.

Overwrite this method if you want to write the report file(s) directly. You should write them to self.outfile.

class lab.reports.filter.FilterReport(**kwargs)[source]

Filter properties files.

This report only applies the given filter and writes a new properties file to the given output destination.

>>> def remove_openstacks(run):
...     return "openstacks" not in run["domain"]
...
>>> from lab.experiment import Experiment
>>> report = FilterReport(filter=remove_openstacks)
>>> exp = Experiment()
>>> exp.add_report(report, outfile="path/to/new/properties")