[Top] | [Contents] | [Index] | [ ? ]

# FATE Automated Testing Environment

# 1. Introduction

FATE provides a regression testsuite embedded within the Libav build system. It can be run locally and optionally configured to send reports to a web aggregator and viewer http://fate.libav.org.

It is advised to run FATE before submitting patches to the current codebase and provide new tests when submitting patches to add additional features.

# 2. Running FATE

## 2.1 Samples and References

In order to run, FATE needs a large amount of data (samples and references) that is provided separately from the actual source distribution.

To inform the build system about the testsuite location, pass ``--samples=<path to the samples>'` to `configure` or set the SAMPLES Make variable or the LIBAV_SAMPLES environment variable to a suitable value.

To use a custom wrapper to run the test, pass ``--target-exec'` to `configure` or set the TARGET_EXEC Make variable.

The dataset is available through `rsync`, is possible to fetch the current sample using the straight rsync command or through a specific Makefile target.

    # rsync -aL rsync://fate-suite.libav.org/fate-suite/ fate-suite

    # make fate-rsync SAMPLES=fate-suite

# 3. Manual Run

FATE regression test can be run through `make`. Specific Makefile targets and Makefile variables are available:

## 3.1 FATE Makefile targets

``fate-list'`

  List all fate/regression test targets.

``fate-rsync'`

  Shortcut to download the fate test samples to the specified testsuite location.

``fate'`

  Run the FATE test suite (requires the fate-suite dataset).

## 3.2 FATE Makefile variables

``V'`

  Verbosity level, can be set to 0, 1 or 2.

  ``0'`

    show just the test arguments

  ``1'`

    show just the command used in the test

  ``2'`

    show everything

``SAMPLES'`

  Specify or override the path to the FATE samples at make time, it has a meaning only while running the regression tests.

``THREADS'`

  Specify how many threads to use while running regression tests, it is quite useful to detect thread-related regressions.

``THREAD_TYPE'`

  Specify which threading strategy test, either slice or frame, by default slice+frame

``CPUFLAGS'`

  Specify a mask to be applied to autodetected CPU flags.

``TARGET_EXEC'`

  Specify or override the wrapper used to run the tests.

``GEN'`

  Set to 1 to generate the missing or mismatched references.

        make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate

# 4. Automated Tests

In order to automatically testing specific configurations, e.g. multiple compilers, `tests/fate.sh` is provided.

This shell script builds Libav, runs the regression tests and prepares a report that can be sent to http://fate.libav.org/ or directly examined locally.

## 4.1 Testing Profiles

The configuration file passed to `fate.sh` is shell scripts as well.

It must provide at least a slot identifier, the repo from which fetch the sources, the samples directory, a workdir with enough space to build and run all the tests. Optional submit command fate_recv and a comment to describe the testing profile are available.

Additional optional parameter to tune the Libav building and reporting process can be passed.

    slot=                                   # some unique identifier
    repo=git://git.libav.org/libav.git      # the source repository
    samples=/path/to/fate/samples
    workdir=                                # directory in which to do all the work
    fate_recv="ssh -T fate@fate.libav.org"  # command to submit report
    comment=                                # optional description
    build_only=     # set to "yes" for a compile-only instance that skips tests

    # the following are optional and map to configure options
    arch=
    cpu=
    cross_prefix=
    as=
    cc=
    ld=
    target_os=
    sysroot=
    target_exec=
    target_path=
    target_samples=
    extra_cflags=
    extra_ldflags=
    extra_libs=
    extra_conf=     # extra configure options not covered above

    #make=          # name of GNU make if not 'make'
    makeopts=       # extra options passed to 'make'
    #tar=           # command to create a tar archive from its arguments on
                    # stdout, defaults to 'tar c'

## 4.2 Special Instances

The TARGET_EXEC option provides a way to run FATE wrapped in `valgrind`, `qemu-user` or `wine` or on remote targets through `ssh`.

## 4.3 Submitting Reports

In order to send reports you need to create an `ssh` key and send it to root@libav.org. The current server fingerprint is a4:99:d7:d3:1c:92:0d:56:d6:d5:61:be:01:ae:7d:e6

[Top] | [Contents] | [Index] | [ ? ]

# Table of Contents

[Top] | [Contents] | [Index] | [ ? ]

# About This Document

This document was generated on June, 28 2015 using texi2html 1.76.

The buttons in the navigation panels have the following meaning:

Button | Name | Go to | From 1.2.3 go to
--- | --- | --- | ---
[ < ] | Back | previous section in reading order | 1.2.2
[ > ] | Forward | next section in reading order | 1.2.4
[ << ] | FastBack | beginning of this chapter or previous chapter | 1
[ Up ] | Up | up section | 1.2
[ >> ] | FastForward | next chapter | 2
[Top] | Top | cover (top) of document |
[Contents] | Contents | table of contents |
[Index] | Index | index |
[ ? ] | About | about (help) |

where the Example assumes that the current position is at Subsubsection One-Two-Three of a document of the following structure:

- 1. Section One
  - 1.1 Subsection One-One
    - ...
  - 1.2 Subsection One-Two
    - 1.2.1 Subsubsection One-Two-One
    - 1.2.2 Subsubsection One-Two-Two
    - 1.2.3 Subsubsection One-Two-Three <== Current Position
    - 1.2.4 Subsubsection One-Two-Four
  - 1.3 Subsection One-Three
    - ...
  - 1.4 Subsection One-Four