It seems that the beta branch failures for the raptor browsertime benchmark tests on the Linux and MacOS platforms are working again.
It is currently unknown why these tests were perma failing at the time. Bug 1832290 has been filed to look into such related issues.
Differential Revision: https://phabricator.services.mozilla.com/D177674
This patch adds documentation to the BasePerfComparator, and also adds some documentation for Comparators in general to the mach-try-perf Firefox source docs.
Differential Revision: https://phabricator.services.mozilla.com/D177760
This patch extends the chromium-as-release perftest configuration to be able to also run on the Windows platform using a Windows build.
To start with, it will only run a subset of priority benchmarks (e.g. sp/sp3)
Differential Revision: https://phabricator.services.mozilla.com/D175788
Previously, the logic for profling raptor tests was intended for browsertime pageload tests. The profiles for benchmark tests were not that useful.
This patch uses a new command in browsertime which makes use of the exposed geckoprofiler start/stop commands to manually choose when to start and stop browsertime through our own custom scripts.
Differential Revision: https://phabricator.services.mozilla.com/D175793
This patch adds python support classes for tests. With them, we'll be able to customize the results, and output parsing better, and with less impact on other existing tests. The first method available allows a test author to modify the command before the test runs. A sample test is also added to show how it all works. No CI tests will be added for it.
Differential Revision: https://phabricator.services.mozilla.com/D173870
This patch lays a foundation for running chromium-as-release binaries with our raptor-browsertime tests.
To start with, this just deals with the Linux platform and minimal changes will be required in patches in the near future for Win/MacOS platforms.
To begin with, we only test on priority benchmarks at the moment (e.g. sp/sp3).
Differential Revision: https://phabricator.services.mozilla.com/D173757
This patch adds three new shippable platforms running the nightly-as-release build. At the same time, browsertime essential tests are enabled for all three platforms on autoland.
Depends on D172029
Differential Revision: https://phabricator.services.mozilla.com/D172030
This patch lays a foundation for running chromium-as-release binaries with our raptor-browsertime tests.
To start with, this just deals with the Linux platform and minimal changes will be required in patches in the near future for Win/MacOS platforms.
To begin with, we only test on priority benchmarks at the moment (e.g. sp/sp3).
Differential Revision: https://phabricator.services.mozilla.com/D173757
There is a perma failure for browsertime benchmark tests affecting mozilla-beta branch. It likely will affect mozilla-release as well.
This patch temporarily disables the tests from running on these branches.
Differential Revision: https://phabricator.services.mozilla.com/D175058
This patch adds three new shippable platforms running the nightly-as-release build. At the same time, browsertime essential tests are enabled for all three platforms on autoland.
Depends on D172029
Differential Revision: https://phabricator.services.mozilla.com/D172030
Temporarily disable the live imgur page load tests on all platforms, for the Chrome application.
This test seems to be permafailing since 2023-03-01.
Differential Revision: https://phabricator.services.mozilla.com/D173982
We have evidence that http/2 upload speed is slow on fast networks.
Adding a test so that we can tune this on all plaforms
Differential Revision: https://phabricator.services.mozilla.com/D172467
This patch adds three new shippable platforms running the nightly-as-release build. At the same time, browsertime essential tests are enabled for all three platforms on autoland.
Depends on D172029
Differential Revision: https://phabricator.services.mozilla.com/D172030
This patch should disable fenix/geckoview *fission* performance tests only.
The tests are still available to select through Try.
Differential Revision: https://phabricator.services.mozilla.com/D171133
flexbox emulation has been on by default in the browser for a while now
(since 108), so these tests are not useful anymore.
Depends on D170944
Differential Revision: https://phabricator.services.mozilla.com/D170945
This patch adds a new test for our regression detection system. It produces a single constant value that we can easily change in the test manifest to trigger an alert. This is being done to in preparation for work on the entire alerting system, and to test if some unimplemented fields in the harness work (min_back_window, max_back_window, and fore_window). They all exist in the performance artifact schemas in mozilla-central, and the treeherder repo. I've also disabled the push/backstop optimizations for the regression tests so that they run on each push. It's limited to running on linux machines.
Differential Revision: https://phabricator.services.mozilla.com/D168116
This patch attempts to address a high frequency of failing imgur raptor-browsertime tests on macos.
Additionally, the secondary url is replaced as it gives a 404
Differential Revision: https://phabricator.services.mozilla.com/D168106
This patch adds an FAQ (Frequently Asked Questions) section to the mach try perf docs. It also does a small cleanup to move fxrecord into the `testing/performance` folder, re-organize the linting configuration file, fix file naming, and captializes the `mozperftest` and `fxrecord` title names in the side-bar. Lastly, it adds a warning to the `mozperftest` docs to direct people who make it there to the `mach try perf` page.
Differential Revision: https://phabricator.services.mozilla.com/D167555
This patch updates the S3 commit used, fixes an issue with local Windows benchmark runs, and enables the windows S3 tests in CI.
Differential Revision: https://phabricator.services.mozilla.com/D166323
