Commit Graph

179 Commits

Author SHA1 Message Date
Dimi Lee
ccd7628134 Bug 1407123 - Wait for the update task to be finished before running next test case. r=francois
The test_partialUpdateV4() test case doesn't wait for the update task
to be finished. It checks the status in the HTTP server side and then calls
run_next_test(). However, when XPCShell test is done, it will trigger
the shutdown process and hence interrupt the ongoing update task.

This cause the xpcshell test receives an error since the update is
interrupted and returns an error like NS_ERROR_UC_UPDATE_SHUTDOWNING.

This patch also fixes a javascript error that we didn't stop the httpd
server when cleanup.

Differential Revision: https://phabricator.services.mozilla.com/D2360
2018-07-26 18:26:19 +00:00
Francois Marier
e4cf2350e6 Bug 1452445 - Fix update error observer in unit test. r=gcp
We should only observe for update errors while we are expecting
a successful update.

MozReview-Commit-ID: 3grGhmxqhIX
2018-05-29 18:06:49 -07:00
Bogdan Tara
d3d6209bd7 Backed out changeset 99f438f07d89 (bug 1452445) for es lint failure on head_urlclassifier.js:440 CLOSED TREE 2018-06-07 01:46:07 +03:00
Francois Marier
26e8ca9ec6 Bug 1452445 - Fix update error observer in unit test. r=gcp
We should only observe for update errors while we are expecting
a successful update.

MozReview-Commit-ID: 3grGhmxqhIX
2018-05-29 18:06:49 -07:00
Kris Maglione
c03fa450d6 Bug 1460092: Add ESLint rule to enforce use of ChromeUtils.generateQI. r=Gijs
Also fixes existing code which fails the rule.

MozReview-Commit-ID: CkLFgsspGMU
2018-05-08 18:36:22 -07:00
Francois Marier
9bca40d5ff Bug 1438671 - Terminate tests early when updates fail. r=gcp
Some of the tests don't handle update errors very well and rely on timeouts to
fire after several minutes. Since these tests are not actually testing update
failure modes, it's safe to fail quicly and terminate the test with an
exception.

MozReview-Commit-ID: EJgaWke6kl2
2018-03-27 16:29:02 -07:00
Francois Marier
f890efb3fa Bug 1436213 - Make test_bug1274685_unowned_list.js work on pver2 and pver4. r=gcp
This test is supposed to verify that Safe Browsing providers can
be initialized correctly even when a table is not configured
properly.

By removing a table from both google and google4, we ensure that
the test will be meaningful regardless of the stack in use.

Also filter out the console noise triggered by looking for the
update and gethash URLs of the "test" dummy provider.

MozReview-Commit-ID: KjWqSqA4FxJ
2018-02-06 15:36:48 -08:00
Andrew McCreight
272cee1e65 Bug 1432992, part 1 - Remove definitions of Ci, Cr, Cc, and Cu. r=florian
This patch was autogenerated by my decomponents.py

It covers almost every file with the extension js, jsm, html, py,
xhtml, or xul.

It removes blank lines after removed lines, when the removed lines are
preceded by either blank lines or the start of a new block. The "start
of a new block" is defined fairly hackily: either the line starts with
//, ends with */, ends with {, <![CDATA[, """ or '''. The first two
cover comments, the third one covers JS, the fourth covers JS embedded
in XUL, and the final two cover JS embedded in Python. This also
applies if the removed line was the first line of the file.

It covers the pattern matching cases like "var {classes: Cc,
interfaces: Ci, utils: Cu, results: Cr} = Components;". It'll remove
the entire thing if they are all either Ci, Cr, Cc or Cu, or it will
remove the appropriate ones and leave the residue behind. If there's
only one behind, then it will turn it into a normal, non-pattern
matching variable definition. (For instance, "const { classes: Cc,
Constructor: CC, interfaces: Ci, utils: Cu } = Components" becomes
"const CC = Components.Constructor".)

MozReview-Commit-ID: DeSHcClQ7cG
2018-02-06 09:36:57 -08:00
Florian Quèze
c5f15a4700 Bug 1339461 - script-generated patch to convert foo.indexOf(...) == -1 to foo.includes(), r=Mossop. 2018-02-01 20:45:22 +01:00
Kris Maglione
0bb74efdf1 Bug 1431533: Part 5a - Auto-rewrite code to use ChromeUtils import methods. r=florian
This was done using the following script:
37e3803c7a/processors/chromeutils-import.jsm

MozReview-Commit-ID: 1Nc3XDu0wGl
2018-01-29 15:20:18 -08:00
Cosmin Sabou
f09d6d985d Backed out 3 changesets (bug 1431533) for Android mochitest failures on testEventDispatcher on a CLOSED TREE
Backed out changeset a1eca62826a1 (bug 1431533)
Backed out changeset 34c999fa006b (bug 1431533)
Backed out changeset e2674287e57f (bug 1431533)
2018-01-30 07:17:48 +02:00
Kris Maglione
fd67f090b2 Bug 1431533: Part 5a - Auto-rewrite code to use ChromeUtils import methods. r=florian
This was done using the following script:
37e3803c7a/processors/chromeutils-import.jsm

MozReview-Commit-ID: 1Nc3XDu0wGl
2018-01-29 15:20:18 -08:00
Brindusan Cristian
483ba301cb Backed out 2 changesets (bug 1431533) for ESlint failures on a CLOSED TREE
Backed out changeset 6e56f4c8843e (bug 1431533)
Backed out changeset 12fc4dee861c (bug 1431533)
2018-01-30 02:32:43 +02:00
Kris Maglione
683a97d172 Bug 1431533: Part 5a - Auto-rewrite code to use ChromeUtils import methods. r=florian
This was done using the following script:
37e3803c7a/processors/chromeutils-import.jsm

MozReview-Commit-ID: 1Nc3XDu0wGl
2018-01-29 15:20:18 -08:00
Florian Quèze
1838aa9e08 Bug 1421992 - script-generated patch to replace do_execute_soon, do_print and do_register_cleanup with executeSoon, info and registerCleanupFunction, rs=Gijs. 2017-12-21 11:10:23 +01:00
Florian Quèze
082ca8b2a3 Bug 1421992 - script-generated patch to replace do_check_* functions with their Assert.* equivalents, rs=Gijs. 2017-12-21 11:08:17 +01:00
DimiL
47c5b9727a Bug 1421500 - Disable back-off for SafeBrowsing testcases not testing back-off. r=francois
After this patch landed, we will have 3 cases:
1. For providers are not "test", for example, google, google4, mozilla ... etc
   backoff CANNOT be disabled.
2. For "test" provider, if preference "browser.safebrowsing.provider.test.disableBackoff" is ON
   backoff is disabled.
3. For "test" provider, if preference "browser.safebrowsing.provider.test.disableBackoff" is Off
   backoff is NOT disabled.

So if your testcase will use listmanager or hashcompleter, you should try to use "test" provider
if possible, otherwise testcase may encounter intermittent failure due to backoff.

MozReview-Commit-ID: 3BDxs0ARyQM
2017-12-08 12:18:34 +08:00
Csoregi Natalia
7828fc1a71 Backed out changeset 2467ce87767e (bug 1421500) for ESLint failure /toolkit/components/url-classifier/nsUrlClassifierLib.js:108. r=backout on a CLOSED TREE 2017-12-14 04:38:07 +02:00
DimiL
8971ba69f6 Bug 1421500 - Disable back-off for SafeBrowsing testcases not testing back-off. r=francois
After this patch landed, we will have 3 cases:
1. For providers are not "test", for example, google, google4, mozilla ... etc
   backoff CANNOT be disabled.
2. For "test" provider, if preference "browser.safebrowsing.provider.test.disableBackoff" is ON
   backoff is disabled.
3. For "test" provider, if preference "browser.safebrowsing.provider.test.disableBackoff" is Off
   backoff is NOT disabled.

So if your testcase will use listmanager or hashcompleter, you should try to use "test" provider
if possible, otherwise testcase may encounter intermittent failure due to backoff.

MozReview-Commit-ID: 3BDxs0ARyQM
2017-12-08 12:18:34 +08:00
Mark Banner
c6cb2b825a Bug 1412893 - Change instances of using getService to Services.jsm where possible in toolkit/components/url-classifier r=francois
MozReview-Commit-ID: 7ejzcckcwQ1
2017-10-31 20:22:58 +00:00
Mark Banner
e47ccbfec3 Bug 1411368 - Automatically fix no-multi-spaces issues raised when using ESLint 4. r=mossop
MozReview-Commit-ID: H5YVp3rnzGo
2017-10-26 11:47:01 +01:00
Dan Banner
5af4450327 Bug 1408777 - Automatically fix instances of missing semicolons in the tree. r=Standard8
MozReview-Commit-ID: Jm8BRgt6mIv
2017-10-15 20:50:39 +01:00
Thomas Nguyen
141b95e424 Bug 1389315 - Isolate Safe Browsing completions cached by each provider. r=francois
MozReview-Commit-ID: INiuM5ydv2f
2017-09-14 17:05:52 +08:00
Thomas Nguyen
71a001d62e Bug 1394031 - Remove mCryptoHash members of nsUrlClassifierDBServiceWorker and ProtocolParser
The usage of cryptoHash consists of a complete set of Init, Update, and Finish, there's
no reason to keep it around

MozReview-Commit-ID: 7bT9IsWEM5m
2017-09-14 10:51:55 +08:00
Andrew McCreight
ed0e2f0913 Bug 1186409 - Use Cu.getGlobalForObject when importing properties off a JSM global. r=kmag
With JSM global sharing, the object returned by Cu.import() is a
NonSyntacticVariablesObject, rather than a global. Various code tries
to use properties from a JSM global via an import.

Cu.importGlobalProperties can also be used in some places.

MozReview-Commit-ID: HudCXO2GKN0
2017-06-23 13:51:14 -07:00
Thomas Nguyen
8496e0fafa Bug 1385609 - Fix backoff issue that makes SB lists no longer update r=francois
The issue occurs when nsITimer is fired earlier than the backoff time. In that
case, the update doesn't proceed and we never make another attempt because the
backoff update timer was oneshot.

We fix the issue in two ways:

- Add a tolerance of 1 second in case the timer fires too early.
- Set another oneshot timer whenever we are prevented from updating due to
backoff.

MozReview-Commit-ID: E2ogNRsHJVK
2017-08-30 18:04:10 +08:00
Sebastian Hengst
061e6ef724 Backed out changeset 14b4af261515 (bug 1385609) for eslint failures in nsUrlClassifierLib.js. r=backout 2017-09-05 19:04:43 +02:00
Thomas Nguyen
cf82b0fcf8 Bug 1385609 - Fix backoff issue that makes SB lists no longer update r=francois
The issue occurs when nsITimer is fired earlier than the backoff time. In that
case, the update doesn't proceed and we never make another attempt because the
backoff update timer was oneshot.

We fix the issue in two ways:

- Add a tolerance of 1 second in case the timer fires too early.
- Set another oneshot timer whenever we are prevented from updating due to
backoff.

MozReview-Commit-ID: E2ogNRsHJVK
2017-08-30 18:04:10 +08:00
Francois Marier
76d985fb87 Bug 1395733 - Ensure the default value of urlclassifier.malwareTable is always sorted. r=gcp
MozReview-Commit-ID: 10M2ZpDyAI9
2017-08-31 14:54:11 -07:00
Henry Chang
0d7d639fec Bug 1392204 - Disable goog-badbinurl-proto and goog-downloadwhite-proto on Fennec. r=francois
Version 4 of the Google Safe Browsing server will return a 404 if any of the
application reputation lists are requested on Android. As a result, we should
avoid these threat types from being sent along with ANDROID_PLATFORM.

MozReview-Commit-ID: 6TUBVxe455y
2017-08-22 12:20:57 +08:00
Henry Chang
d022ca7e9b Bug 1388582 - Skip unsupported threat types on current platform while making v4 request. r=francois
MozReview-Commit-ID: FTmOMQ339ZL
2017-08-16 15:30:20 +08:00
Dan Banner
0d33a246ae Bug 1386684 - Enable ESLint for toolkit/components/url-classifier (manual fixes). r=hchang
MozReview-Commit-ID: JH1XDFMVd43
2017-08-02 20:09:14 +01:00
Dan Banner
1a5db31879 Bug 1386684 - Enable ESLint for toolkit/components/url-classifier (automatic fixes). r=hchang
MozReview-Commit-ID: F0Z8dRaYOku
2017-08-02 16:12:07 +01:00
Henry Chang
53064b6aa0 Bug 1375277 - Test case for new error code NS_ERROR_HARMFUL_URI. r=francois
The test is based on "test-harmful-simple" so we can add testing entries
in a human-readable way.

MozReview-Commit-ID: IJcYrp04uFs
2017-07-28 17:38:01 +08:00
Florian Quèze
c0468168f5 Bug 1368456 - remove Promise.jsm imports in tests, r=mconley. 2017-06-23 11:25:52 +02:00
DimiL
02e313d384 Bug 1359299 - V4 caches in LookupCache need to be copied around in copy constructor. r=hchang
MozReview-Commit-ID: AjzUUmQKiPW
2017-06-06 14:16:57 +08:00
Ryan VanderMeulen
c121499332 Backed out changeset c0b940487708 (bug 1359299) for causing intermittent Windows safebrowsing crashes. 2017-05-24 09:11:04 -04:00
DimiL
ecf67ffc51 Bug 1359299 - Copy fullhash cache when update. r=hchang
After adopting the new thread model for safebrowsing, we will create a new
lookup cache for update so we can still check lookup cache at the same time.

Prefix set, completions will be generated when we open the new lookup cache
but it won't include cache, so we will loss cache after that.

This patch will copy cache data from old lookup cache to new lookup
cache while update.

MozReview-Commit-ID: L0WpiHOGIGm
2017-05-23 09:19:06 +08:00
Thomas Nguyen
379b129513 Bug 1363879 - Sort gethash prefixes to hide noise entries r=francois
In V2 we shuffled the hash entries before sending the request to obscure the real
entry from noises. We should also hide the real entry in V4. Using sort() is
enough for both V2 and V4 because the array contains exactly 5 entries in almost
all cases.

MozReview-Commit-ID: 4uOXIF83KQL
2017-05-17 16:11:36 +08:00
DimiL
a0b8501692 Bug 1333328 - Refactor cache miss handling mechanism for V2. r=francois
In this patch, we will make Safebrowsing V2 caching use the same algorithm as V4.
So we remove "mMissCache" for negative caching and TableFresness check for
positive caching.

But Safebrowsing V2 doesn't contain negative/positive cache duration information in
gethash response. So we hard-code a fixed value, 15 minutes, as cache duration.
In this way, we can sync the mechanism we handle caching for V2 and V4.

An extra effort for V2 here is that we need to manually record prefixes misses
because we won't get any response for those prefixes(implemented in
nsUrlClassifierLookupCallback::CacheMisses).
2017-05-04 09:38:14 +08:00
Florian Quèze
15feaa278a Bug 1362882 - script-generated patch to switch from Promise.defer() to new Promise() where it's straightforward, rs=Mossop. 2017-05-12 14:55:27 +02:00
dimi
5b1a0ff5b5 Bug 1311935 - P2. Process fullHashes.find response. r=francois
This patch includes following changes:

1. nsUrlClassifierHashCompleter.js
   nsUrlClassifierHashCompleter.idl
   - Add completionV4 interface for hashCompleter to pass response data to
     DB service.
   - Process response data includes negative cache duration, matched full
     hashes and cache duration for each match. Full matches are passed through
     nsIFullHashMatch interface.
   - Change _requests.responses from array contains matched fullhashes to
     dictionary so that it can store additional information likes negative cache
     duration.
2. nsUrlClassifierDBService.cpp
   - Implement CompletionV4 interface, store response data to CacheResultV4
     object. Expired duration to expired time is handled here.
   - Add CacheResultToTableUpdate function to convert V2 & V4 cache result
     to TableUpdate object.
3. LookupCache.h
   - Extend CacheResult to CacheResultV2 and CacheResultV4 so we can store
     response data in CompletionV2 and CompletionV4.
4. HashStore.h
   - Add API and member variable in TableUpdateV4 to store response data.
     TableUpdate object is used by DB service to pass update data or gethash
     response to Classifier, so we need to extend TableUpdateV4 to be able
     to store fullHashes.find response.
6. Entry.h
   - Define the structure about how we cache fullHashes.find response.

MozReview-Commit-ID: FV4yAl2SAc6
2017-04-11 11:50:48 +08:00
Iris Hsiao
cd018fd494 Backed out 4 changesets (bug 1311935) for causing assertion crash by developer's request
Backed out changeset 27e624cd9479 (bug 1311935)
Backed out changeset 4c0381ab0990 (bug 1311935)
Backed out changeset 73587838ef16 (bug 1311935)
Backed out changeset a5a6c0f79733 (bug 1311935)
2017-04-11 11:04:54 +08:00
dimi
5856f89d1f Bug 1311935 - P2. Process fullHashes.find response. r=francois
This patch includes following changes:

1. nsUrlClassifierHashCompleter.js
   nsUrlClassifierHashCompleter.idl
   - Add completionV4 interface for hashCompleter to pass response data to
     DB service.
   - Process response data includes negative cache duration, matched full
     hashes and cache duration for each match. Full matches are passed through
     nsIFullHashMatch interface.
   - Change _requests.responses from array contains matched fullhashes to
     dictionary so that it can store additional information likes negative cache
     duration.
2. nsUrlClassifierDBService.cpp
   - Implement CompletionV4 interface, store response data to CacheResultV4
     object. Expired duration to expired time is handled here.
   - Add CacheResultToTableUpdate function to convert V2 & V4 cache result
     to TableUpdate object.
3. LookupCache.h
   - Extend CacheResult to CacheResultV2 and CacheResultV4 so we can store
     response data in CompletionV2 and CompletionV4.
4. HashStore.h
   - Add API and member variable in TableUpdateV4 to store response data.
     TableUpdate object is used by DB service to pass update data or gethash
     response to Classifier, so we need to extend TableUpdateV4 to be able
     to store fullHashes.find response.
6. Entry.h
   - Define the structure about how we cache fullHashes.find response.

MozReview-Commit-ID: KgR1NASl7GC
2017-04-10 16:20:09 +08:00
Henry Chang
4543c61a48 Bug 1339050 - Asynchronously apply safebrowsing DB update. r=francois,gcp
A new function Classifier::AsyncApplyUpdates() is implemented for async update.
Besides, all public Classifier interfaces become "worker thread only" and
we remove DBServiceWorker::ApplyUpdatesBackground/Foreground.

In DBServiceWorker::FinishUpdate, instead of calling Classifier::ApplyUpdates,
we call Classifier::AsyncApplyUpdates and install a callback for notifying
the update observer when update is finished. The callback will occur on the
caller thread (i.e. worker thread.)

As for the shutdown issue, when the main thread is notified to shut down,
we at first *synchronously* dispatch an event to the worker thread to
shut down the update thread. After getting synchronized with all other
threads, we send last two events "CancelUpdate" and "CloseDb" to notify
dangling update (i.e. BeginUpdate is called but FinishUpdate isn't)
and do cleanup work.

MozReview-Commit-ID: DXZvA2eFKlc
2017-04-06 07:07:56 +08:00
Mark Banner
ec9074e132 Bug 1347712 - Change the testing configurations into ESLint configurations within eslint-plugin-mozilla - automatically update .eslintrc.js test config files for new config locations. r=jaws
MozReview-Commit-ID: LH0CQOPfoe6
2017-03-20 12:36:37 +01:00
Henry Chang
b1b06f3fe9 Bug 1346757 - Change the downloadError callback timing. r=francois
MozReview-Commit-ID: JleLPltEBOw
2017-03-13 21:16:07 +08:00
Thomas Nguyen
96756deadd Bug 1329558 - Implement Minimum wait duration for V4 gethash r=francois
MozReview-Commit-ID: 7i9Wz7pq0yJ
2017-02-09 16:56:44 +08:00
Mark Banner
55868dde7c Bug 503613 - Remove old 'tail =' lines from xpcshell.ini files; r=gps
MozReview-Commit-ID: 62Hp5ISxowJ
2017-01-18 10:30:39 +00:00
Mark Banner
f7889a82eb Bug 527444 - use do_register_cleanup calls instead of tail_*.js files in xpcshell, in Core and Toolkit. r=Paolo.
Patch originally by Rahid Hasan. Updated by Akshendra Pratap Singh, jdm and Standard8.

MozReview-Commit-ID: F0edTCSsXzy
2017-01-12 16:09:58 +00:00