Design Specification for ./mach manifest skip-fails
===================================================

The first of the mach manifest subcommands is skip-fails. This command
can be used to automatically edit manifests to skip tests that are
failing as well as file the corresponding bugs for the failures. This
is particularly useful when “greening up” a new platform.

The user documentation for skip-fails is here:
https://firefox-source-docs.mozilla.org/mozbase/manifestparser.html#using-mach-manifest-skip-fails

The skip-fails command knows how to manage failures for TOML, WPT
and REFTEST manifests. Additional developer design notes are included here.


Running skip-fails for TOML
---------------------------

The user documentation for TOML manifests is:
https://firefox-source-docs.mozilla.org/mozbase/manifestparser.html
https://firefox-source-docs.mozilla.org/testing/tests-for-new-config/index.html#run-skip-fails

Usage
-----

$ ./mach manifest skip-fails -h
usage: mach [global arguments] manifest skip-fails [command arguments]

Global Arguments:

  -v, --verbose         Print verbose output.
  --quiet               Don't print as much output.
  -l, --log-file FILENAME
                        Filename to write log data to.
  --log-interval        Prefix log line with interval from last message rather
                        than relative time. Note that this is NOT execution
                        time if there are parallel operations.
  --no-interactive      Automatically selects the default option on any
                        interactive prompts. If the output is not a terminal,
                        then --no-interactive is assumed.
  --log-no-times        Do not prefix log lines with times. By default, mach
                        will prefix each output line with the time since
                        command start.
  -h, --help            Show this help message.
  --debug-command       Start a Python debugger when command is dispatched.
  --profile-command     Capture a Python profile of the mach process as command
                        is dispatched.
  --settings FILENAME   Path to settings file.

Sub Command Arguments:
  try_url               Treeherder URL for try (please use quotes)
  -b, --bugzilla BUGZILLA
                        Bugzilla instance [disable]
  -C, --clear-cache [CLEAR_CACHE]
                        clear cache REVISION (or all)
  -c, --carryover       Set carryover mode (only skip failures for platform
                        matches)
  -d, --dry-run         Determine manifest changes, but do not write them
  -F, --use-failures USE_FAILURES
                        Use failures from file
  -f, --save-failures SAVE_FAILURES
                        Save failures to file
  -I, --implicit-vars   Use implicit variables in reftest manifests
  -i, --task-id TASK_ID
                        Task id to write a condition for instead of all tasks
                        from the push
  -k, --known-intermittents
                        Set known intermittents mode (only skip failures known
                        intermittents)
  -M, --max-failures MAX_FAILURES
                        Maximum number of failures to skip (-1 == no limit)
  -m, --meta-bug-id META_BUG_ID
                        Meta Bug id
  -n, --new-version NEW_VERSION
                        New version to use for annotations
  -N, --new-failures    Set new failures mode (only add conditions for new
                        failures)
  -r, --failure-ratio FAILURE_RATIO
                        Ratio of test failures/total to skip [0.4]
  -R, --replace-tbd     Replace Bug TBD in manifests by filing new bugs
  -s, --turbo           Skip all secondary failures
  -T, --use-tasks USE_TASKS
                        Use tasks from file
  -t, --save-tasks SAVE_TASKS
                        Save tasks to file
  -u, --user-agent USER_AGENT
                        User-Agent to use for mozci if queries are forbidden
                        from treeherder
  -v, --verbose         Verbose mode


Design
------

1. Classify the failure based on the following:
   a. If less than 3 runs, classify "unknown"
   b. If zero failures, classify "success"
   c. If the failure ratio of failed runs / total runs is less than 40%
      classify "intermittent"
   d. Else edit the manifest to skip this failure

2. Manifest variable information
   a. Variables used in Python are set:
      https://searchfox.org/firefox-main/source/testing/mozbase/mozinfo/mozinfo/platforminfo.py

      NOTE: that the task metadata runtimes are turned in to test variants by looking
      up the "mozinfo" field that corresponds to the "runtime" key in
      taskcluster/test_configs/variants.yml

      NOTE: special excpetions are these runtimes are mapped to these variants:
      - no-fission -> !fission
      - 1proc -> !e10s

      NOTE: multiple variants are stored in the field test_variant concatenated by '+',
      for example: "socketprocess_networking+!fission"

   b. Variables used in JavaScript are set:
      https://searchfox.org/firefox-main/source/layout/tools/reftest/manifest.sys.mjs

   c. Use arch instead of processor
      https://searchfox.org/firefox-main/source/testing/mozbase/mozinfo/mozinfo/mozinfo.py#170

      aarch64
      arm7
      x86
      x86_64

   d. Build types
      An an optimized build "opt" implies none of these build types: "debug", "ccov", "asan", "tsan".
      A build type "!debug" implies "opt"

   e. A "platform match" for the purpose of bug carryover means
      - "os" is not specified in the existing condition or "os" is the same, and
      - The existing build type is not present and the "os" explictly the same, or
      - The existing build type is present:
        + "opt" (or "!debug") in both existing skip-if and proposed skip-if
        + "!opt" or one of ["debug", "asan", "tsan", "ccov" ] in both
          (covers the case of "debug" in both)
          EXCEPT does not match if one is "debug" and the other is
          in ["asan", "tsan", "ccov"]
      - Else not a match

   f. If the argument `-n, --new-version NEW_VERSION` is used
      then the os_version used in any skip-if expression
      will be NEW_VERSION -- regardless of the os, arch or
      another actual platform information. This is intended to workaround a bug
      where task metadata would report "macosx 10.15", yet the worker was actually
      "macosx 14.70". By setting `--new-version 14.70` then any skip-if
      expression will be created with `os_version == 14.70` which would then match
      what mozinfo says on the next try run.

3. Bugzilla interaction
   a. The bugzilla instance will be set to (in increasing precedence):
      i. Default: "bugzilla.allizom.org"
      ii. From the environment variable BUGZILLA
      iii. From the command line arg --bugzilla

   b. NOTE: if the BUGZILLA env var or command line arg is "disable"
      then Bugzilla interaction will be disabled.

   c. Setting --dry-run prevents editing manifests, but will not disable reading Bugzilla.
      Nevertheless no new bugs will be created nor comments added.
      Note existing bugs may be mentioned in an info message as appropriate.

   d. If --meta-bug-id is NOT set no bugs will be filed. Any created manifest bugs will
      require referencing a meta-bug-id.

   e. If there is a skip line for this manifest and the platform matches (see above) then
          add an info message about carryover
          do NOT edit any existing bug nor create a new bug
          If there is an existing manifest comment, then carry it over to the new skip line
          Else do NOT add a new comment (as a comment did not exist before)
      Else (no skip lines match the platform)
          Add a new skip-if line and reference the bug for this manifest + meta-bug-id
          (create a new bug if needed)

   f. Special case for carryover when a [default] section exists
          and contains a skip-if with a platform match
      AND our current failure matches that skip-if line for a real manifest
      THEN carry over the comment from [default] to the new skip-if line.

4. Modes for skip-fails
   a. Normal mode
      Tests with failures found based on the above heuristics
      are disabled with skip-if conditions in the appropriate manifests
      and bugs filed (or updated).

   b. Carry-over mode
      Only consider adding skip-if conditions which match the
      platform (see above) of previous conditions. Does not perform any
      bugzilla changes. Requires meta-bug-id ( future actions will be cached in
      {topsrcdir}/.skip_fails_cache/meta-bug-{meta_bug_id}/actions.json )

   c. Known intermittents mode
      Only consider adding skip-if conditions which have known
      intermittent bugs (and are not carryover bugs).
      form (see above) of previous conditions. Does not perform any
      bugzilla changes. Requires meta-bug-id ( future actions will be cached in
      {topsrcdir}/.skip_fails_cache/meta-bug-{meta_bug_id}/actions.json )

      For each failure, if the job_id can be determined then bug_suggestions will
      be retrieved from
      https://treeherder.mozilla.org/api/project/{repo}/jobs/{job_id}/bug_suggestions/
      each object with "search" field of ^PROCESS-CRASH or ^TEST-UNEXPECTED
      will be considered if one of the bugs in
      bugs.open_recent.<i>.summary ends with 'single tracking bug$'
      then
      bugs.open_recent.<i>.id has the bug id

   d. New failures mode
      Will only edit manifest skip-if conditions for new failures
      (i.e. not carryover nor known intermittents). Will record
      bug as "TBD". Requires meta-bug-id ( future actions will be cached in
      {topsrcdir}/.skip_fails_cache/meta-bug-{meta_bug_id}/actions.json )

   e. Replace TBD mode
      Will only edit manifest skip-if conditions for new failures
      by filing new bugs and replacing TBD with actual bug number.
      Requires meta-bug-id ( new bug actions will be read from
      {topsrcdir}/.skip_fails_cache/meta-bug-{meta_bug_id}/actions.json )

5. Cache for skip-fails
   At the top of the source tree is a cache directory for skip-fails:
   ".skip_fails_cache". This directory will be created if it is not present.
   When running skip-fails if --use-tasks has not been set then
   the cache directory will be searched for a tasks file corresponding
   the revision (i.e. ".skip_fails_cache/REVISION/tasks.json"). If
   that file is present then it is used. Similarly for --use-failures.
   If --save-tasks has not been set then, by default, the tasks will
   be saved in the cache directory. Similarly for --save-failures.
   Any specific cache directory can be cleared with --clear-cache REVISION.
   All cache directories can be deleted with --clear-cache (no argument).
   Any cache directories which are over 45 days old will be automatically
   cleared.

As design changes for skip-fails / TOML, notes will be added here.


Running skip-fails for WPT
---------------------------

The user documentation for WPT manifests is:
https://firefox-source-docs.mozilla.org/web-platform/index.html

1. Classify the failure based on the following:
   a. If less than 3 runs, classify "unknown"
   b. If zero failures, classify "success"
   c. If the ratio of failed runs / total runs is less than 40%
      classify "intermittent"
   d. Else edit the manifest to skip this failure

As design changes for skip-fails / WPT, notes will be added here.


Running skip-fails for REFTEST
------------------------------

The user documentation for REFTEST manifests is:
https://firefox-source-docs.mozilla.org/layout/Reftest.html

Current Design notes for skip-fails / REFTEST:

As skip-fails for REFTEST finds a failure it will take the following actions:

1. Classify the failure based on the following:
   a. If less than 3 runs, classify "unknown"
   b. If zero failures, classify "success"
   c. If the ratio of failed runs / total runs is less than 40%
      classify "low frequency intermittent" (INTERMITTENT)
   c. If the ratio of failed runs / total runs is less than 80%
      classify "high frequency intermittent" (DISABLE_INTERMITTENT)
   d. If the ratio is <= 100%, classify "failure" (DISABLE_FAILURE)

2. Propose a new range (differences, pixels)
   a. For "low frequency intermittent" errors do nothing
   b. For "high frequency intermittent" set the range 0-Y (i.e. always
      start at zero to avoid UNEXPECTED PASS)
   c. For "failure" set the range X-Y (i.e. the range as reported in
      reftest_errorsummary.log). If the failure is UNEXPECTED PASS
      then set the range to 0-Y.
   d. NOTE: The maximum for differences and pixels will be increased
      by 5% over the reported values in reftest_errorsummary.log
   e. NOTE: If for whatever reason the range is calculated to be 0-0 then
      remove the fuzzy-if entirely

3. Merge with an existing fuzzy-if for this OS platform if the other
   criteria differ in only one dimension. Current dimensions include:

   OS: gtkWidget, winWidget, cocoaWidget, Android
   build_type: optimized, isDebugBuild, isCoverageBuild, AddressSanitizer, ThreadSanitizer
   is64Bit
   fission
   useDrawSnapshot
   swgl
   <OTHER>

4. Based on a command line switch choose do use implicit variables (or not).
   a. The following variables can implicit values:
      is64Bit: true
      fission: true
      useDrawSnapshot: false
      swgl: false
      If the value of an implicit variable matches it's default then it may be
      elided from the conditional expression.
      If the value of an implicit variable is the oppposite of it's default
      value then it may be specified in the condition, e.g. !is64Bit
      When merging an existing condition with a new failure condition where
      the one dimension of difference is an implicit variable then the
      condition must include the disjunction of the variable, e.g. (is64Bit&&!is64Bit)
      That is essential so that the reftest sandbox does not add the implicit
      variable and cover only one of the two possible values.
   b. When implicit variables are not used then the following heuristics apply:
      The variable is64Bit will be elided for all platforms except for winWidget.
      The variable useDrawSnapshot will be elided for all platforms except gtkWidget.

5. Position of fuzzy-if conditions
   In reftest manifest the rightmost applicable condition is used.

   a. First sorting criterion is based on OS in this order:
      OSES = ["Android", "cocoaWidget", "appleSilicon", "gtkWidget", "winWidget"]
      This is because appleSilicon is a narrower specification than
      cocoaWidget and would be ignored if not to the right of cocoaWidget.

   b. The second sorting criterion is the number of dimensions that participate
      in the expression (including implicits) from least dimensions to
      greatest dimensions.

   The last fuzzy-if will be placed on the line immediately to the left of the
   "type" (e.g. == or !=) (or to the left of an optional "HTTP" expression, if present).
   Non fuzzy-if statements will appear before the fuzzy-if statements.
   In this way the more generic conditions will only be superceded by very specific
   conditions.


Design Specification for ./mach manifest clean-skip-fails
=========================================================

The purpose / rationale for this command is to-be-documented.

description="Update manifests to remove skip-if conditions for a specific platform.
             Only works for TOML manifests.",

The user documentation is:
https://firefox-source-docs.mozilla.org/...

Usage
-----
Sub Command Arguments:
  manifest_search_path  Path to the folder containing the manifests to update,
                        or the path to a single manifest
  -o, --os OS_NAME      OS to remove (linux, mac, win)
  -s, --os_version OS_VERSION
                        Version of the OS to remove (eg: 18.04 for linux)
  -p, --processor PROCESSOR
                        Type of processor architecture to remove (eg: x86)


Design Specification for ./mach manifest high-freq-skip-fails
=========================================================

The purpose / rationale for this command is to-be-documented.

description="Update manifests to skip failing tests",

The user documentation is:
https://firefox-source-docs.mozilla.org/...

Usage
-----
Sub Command Arguments:
  -f, --failures FAILURES
                        Minimum number of failures for the bug to be skipped
  -d, --days DAYS       Number of days to look for failures since now
