Commit Graph

126 Commits

Author SHA1 Message Date
dimi
603431b0dd Bug 1408631 - Release SafeBrowsing lookupcache in worker thread while shutdown. r=francois
MozReview-Commit-ID: HuPUyIDFLPX
2017-10-20 10:18:59 +08:00
Thomas Nguyen
71a001d62e Bug 1394031 - Remove mCryptoHash members of nsUrlClassifierDBServiceWorker and ProtocolParser
The usage of cryptoHash consists of a complete set of Init, Update, and Finish, there's
no reason to keep it around

MozReview-Commit-ID: 7bT9IsWEM5m
2017-09-14 10:51:55 +08:00
Eric Rahm
b3a9bd1c4c Bug 1393235 - Fix improper usages of string functions. r=njn
This fixes usages of `Find`, `RFind` and the equality operator that kind of
work right now but will break with the proper type checking of a templatized
version of the string classes.

For `Find` and `RFind` it appears that `nsCString::(R)Find("foo", 0)` calls
were being coerced to the `Find(char*, bool, int, int)` versions. The intent was
probably to just start searching from position zero.

For the equality operator, the type of nullptr is nullptr_t rather than
char(16_t)* so we'd need to add an operator overload that takes nullptr_t. In
this case just using `IsVoid` is probably more appropriate.
2017-08-22 19:30:46 -07:00
Kartikaya Gupta
14424677af Bug 1384233 - Remove SizePrintfMacros.h. r=froydnj
We have a minimum requirement of VS 2015 for Windows builds, which supports
the z length modifier for format specifiers. So we don't need SizePrintfMacros.h
any more, and can just use %zu and friends directly everywhere.

MozReview-Commit-ID: 6s78RvPFMzv
2017-07-26 16:03:57 -04:00
Thomas Nguyen
41442f215a Bug 1378680 - Refactor usage fallible alloc in url-classifier r=francois
MozReview-Commit-ID: HHG1ByeIwZv
2017-07-20 16:20:27 +08:00
Bill McCloskey
ce42826bdf Bug 1372405 - Provide names for all runnables in the tree (r=froydnj)
MozReview-Commit-ID: DKR6ROiHRS7
2017-06-26 14:19:58 -07:00
Carsten "Tomcat" Book
238bf154d5 Backed out changeset 4f6302a98ae4 (bug 1372405) 2017-06-21 13:59:26 +02:00
Bill McCloskey
67e8af4720 Bug 1372405 - Provide names for all runnables in the tree (r=froydnj)
MozReview-Commit-ID: DKR6ROiHRS7
2017-06-20 21:44:11 -07:00
Carsten "Tomcat" Book
bbe9441993 Backed out changeset 9846de3bd954 (bug 1372405) 2017-06-20 08:27:02 +02:00
Bill McCloskey
f69608368b Bug 1372405 - Provide names for all runnables in the tree (r=froydnj)
MozReview-Commit-ID: DKR6ROiHRS7
2017-06-19 22:25:47 -07:00
DimiL
611d1430b5 Bug 1366965 - Remove telemetry that compare SafeBrowsing V2 & V4. r=francois
MozReview-Commit-ID: 7vudFBK3rdp
2017-06-12 11:27:19 +08:00
DimiL
02e313d384 Bug 1359299 - V4 caches in LookupCache need to be copied around in copy constructor. r=hchang
MozReview-Commit-ID: AjzUUmQKiPW
2017-06-06 14:16:57 +08:00
Ryan VanderMeulen
c121499332 Backed out changeset c0b940487708 (bug 1359299) for causing intermittent Windows safebrowsing crashes. 2017-05-24 09:11:04 -04:00
DimiL
ecf67ffc51 Bug 1359299 - Copy fullhash cache when update. r=hchang
After adopting the new thread model for safebrowsing, we will create a new
lookup cache for update so we can still check lookup cache at the same time.

Prefix set, completions will be generated when we open the new lookup cache
but it won't include cache, so we will loss cache after that.

This patch will copy cache data from old lookup cache to new lookup
cache while update.

MozReview-Commit-ID: L0WpiHOGIGm
2017-05-23 09:19:06 +08:00
DimiL
c8f160ac4e Bug 1360480 - about:url-classifier: Cache information. r=francois
MozReview-Commit-ID: 4YXtb2KPgwL
2017-05-17 10:32:33 +08:00
Chung-Sheng Fu
c1c4081a34 Bug 1342900 - Remove usage of nsTArray<nsAutoPtr<>> in Classifier. r=dimi
MozReview-Commit-ID: 2SGPEuEg0lz
2017-05-05 14:33:18 +08:00
DimiL
769f299a32 Bug 1333328 - Remove mTableFreshness and gFreshnessGuarantee. r=francois
Functions and Preference related to mTableFreshness and gFreshnessGuarantee
could be removed since we will no longer require them.

MozReview-Commit-ID: IAa0UHLSQ56
2017-05-04 11:13:21 +08:00
DimiL
a0b8501692 Bug 1333328 - Refactor cache miss handling mechanism for V2. r=francois
In this patch, we will make Safebrowsing V2 caching use the same algorithm as V4.
So we remove "mMissCache" for negative caching and TableFresness check for
positive caching.

But Safebrowsing V2 doesn't contain negative/positive cache duration information in
gethash response. So we hard-code a fixed value, 15 minutes, as cache duration.
In this way, we can sync the mechanism we handle caching for V2 and V4.

An extra effort for V2 here is that we need to manually record prefixes misses
because we won't get any response for those prefixes(implemented in
nsUrlClassifierLookupCallback::CacheMisses).
2017-05-04 09:38:14 +08:00
Michael Layzell
9e9bb6861f Bug 1339537 - Part 6: Pass std::function values tree by const reference instead of by value, r=ehsan
MozReview-Commit-ID: PVAqU2DPs2
2017-04-27 12:44:57 -04:00
Henry Chang
a629121fce Bug 1354968 - Avoid concurrent access of mTableRefreshness. r=francois
mTableRefreshness, a non-thread-safe object, might be accessed on worker thread
and update thread cocurrently. To solve this issue, on update thread we only
insert data to mNewTableRefreshness and merge to mTableRefreshness on
the worker thread later.

MozReview-Commit-ID: 9WgoeYfWVfK
2017-04-11 01:02:42 +08:00
dimi
bb15dc150d Bug 1311935 - P3. Implement safebrowsing v4 caching logic. r=francois
LookupCacheV4::Has implements safebrowsing v4 caching logic.
1. Check if fullhash match any prefix in local database:
  - If not, the URL is safe.
2. Check if prefix is in the cache(prefix is always the first 4-byte of
   the fullhash, Bug 1323953):
  - If not, send fullhash request
3. Check if fullhash is in the positive cache:
  - If fullhash is found and it is not expired, the URL is not safe.
  - If fullhash is found and it is expired, send fullhash request.
4. If fullhash is not found, check negative cache expired time:
  - If negative cache time is not expired, the URL is safe.
  - If negative cache time is expired, send fullhash request.

MozReview-Commit-ID: GRX7CP8ig49
2017-04-10 14:21:08 +08:00
DimiL
c1a8a088db Bug 1311935 - P1. Make ActiveTables() work for safebrowsing v4. r=francois
This patch fixes that Classifier::ActiveTables doesn't return v4 tables.

Classifier::mActiveTablesCache is generated by scanning safebrowsing directory.
We use Classifier::ScanStoreDir to do the work, but it will ignore subdirectory.
Since v4 tables are stored in subdirectory 'google4', mActiveTablesCache doesn't
include v4 tables.

Fix this issue by checking subdirectory recursively in ScanStoreDir.

MozReview-Commit-ID: I6pa6e4bFND
2017-04-07 14:30:11 +08:00
Iris Hsiao
cd018fd494 Backed out 4 changesets (bug 1311935) for causing assertion crash by developer's request
Backed out changeset 27e624cd9479 (bug 1311935)
Backed out changeset 4c0381ab0990 (bug 1311935)
Backed out changeset 73587838ef16 (bug 1311935)
Backed out changeset a5a6c0f79733 (bug 1311935)
2017-04-11 11:04:54 +08:00
dimi
3a7526678a Bug 1311935 - P3. Implement safebrowsing v4 caching logic. r=francois
LookupCacheV4::Has implements safebrowsing v4 caching logic.
1. Check if fullhash match any prefix in local database:
  - If not, the URL is safe.
2. Check if prefix is in the cache(prefix is always the first 4-byte of
   the fullhash, Bug 1323953):
  - If not, send fullhash request
3. Check if fullhash is in the positive cache:
  - If fullhash is found and it is not expired, the URL is not safe.
  - If fullhash is found and it is expired, send fullhash request.
4. If fullhash is not found, check negative cache expired time:
  - If negative cache time is not expired, the URL is safe.
  - If negative cache time is expired, send fullhash request.

MozReview-Commit-ID: GRX7CP8ig49
2017-04-10 14:21:08 +08:00
DimiL
c4fc0215be Bug 1311935 - P1. Make ActiveTables() work for safebrowsing v4. r=francois
This patch fixes that Classifier::ActiveTables doesn't return v4 tables.

Classifier::mActiveTablesCache is generated by scanning safebrowsing directory.
We use Classifier::ScanStoreDir to do the work, but it will ignore subdirectory.
Since v4 tables are stored in subdirectory 'google4', mActiveTablesCache doesn't
include v4 tables.

Fix this issue by checking subdirectory recursively in ScanStoreDir.

MozReview-Commit-ID: I6pa6e4bFND
2017-04-07 14:30:11 +08:00
Henry Chang
4543c61a48 Bug 1339050 - Asynchronously apply safebrowsing DB update. r=francois,gcp
A new function Classifier::AsyncApplyUpdates() is implemented for async update.
Besides, all public Classifier interfaces become "worker thread only" and
we remove DBServiceWorker::ApplyUpdatesBackground/Foreground.

In DBServiceWorker::FinishUpdate, instead of calling Classifier::ApplyUpdates,
we call Classifier::AsyncApplyUpdates and install a callback for notifying
the update observer when update is finished. The callback will occur on the
caller thread (i.e. worker thread.)

As for the shutdown issue, when the main thread is notified to shut down,
we at first *synchronously* dispatch an event to the worker thread to
shut down the update thread. After getting synchronized with all other
threads, we send last two events "CancelUpdate" and "CloseDb" to notify
dangling update (i.e. BeginUpdate is called but FinishUpdate isn't)
and do cleanup work.

MozReview-Commit-ID: DXZvA2eFKlc
2017-04-06 07:07:56 +08:00
dimi
703eec0632 Bug 1338033 - Remove URLCLASSIFIER_PREFIX_MATCH. r=francois
URLCLASSIFIER_PREFIX_MATCH is replaced by URLCLASSIFIER_FULL_MATCH(Bug 1311931)
so it should be removed.

MozReview-Commit-ID: 4F1B07bGwRP
2017-03-22 17:27:11 +08:00
Thomas Nguyen
23421961ee Bug 1343416 - Use LCG when generating noise entries for Safe Browsing completions r=gcp
MozReview-Commit-ID: FxQH2haAcrf
2017-03-09 10:59:52 +08:00
Henry Chang
75d7890222 Bug 1339760 - Split update process to background/foreground and run background on update thread **synchronously**. r=francois,gcp
MozReview-Commit-ID: J0phPC1nWsf
2017-02-22 17:25:26 +08:00
Henry Chang
73fc8b0a2d Bug 1342397 - Properly reset stuff when failing to create LookupCache for update. r=gcp
MozReview-Commit-ID: 1ecI2lFIj2U
2017-02-25 00:22:39 +08:00
Dimi Lee
7545e85763 Bug 1335974 - Add more error codes to URLCLASSIFIER_UPDATE_ERROR. r=francois
MozReview-Commit-ID: AAd6Yu0AuUO
2017-02-22 16:25:04 +08:00
Thomas Nguyen
2b6e2edd11 Bug 1297962 - Add noise data when sending v4 gethash request r=francois
MozReview-Commit-ID: GbyvX7wcg8c
* * *
[mq]: 1297962_review

MozReview-Commit-ID: 1U2T0wq778R
2017-02-24 10:22:12 +08:00
Dimi Lee
5375a9f0e5 Bug 1311931 - Add telemetry to measure full match rate for v2 and v4. r=francois
MozReview-Commit-ID: H9jAR82rgDh
2017-02-23 23:07:13 +08:00
Henry Chang
c16ea9af44 Bug 1338970 - Introduce 2-step DB update scheme. (background update then swap in) r=francois,gcp
MozReview-Commit-ID: 4C9N0tuSso0
2017-02-21 14:18:29 +08:00
Tom Tromey
a4b717ab39 Bug 1060419 - make log_print use Printf.h, r=froydnj
MozReview-Commit-ID: BIZ1GQEZ1vs
2016-12-15 20:16:31 -07:00
dimi
5750b4baba Bug 1336909 - Restrict URLCLASSIFIER_PREFIX_MATCH to profiles that have working V4. r=francois
MozReview-Commit-ID: L3lKgiohalH
2017-02-08 15:18:35 +08:00
Dimi Lee
45dd73df16 Bug 1333257 - Only cache V2 misses when doing Safe Browsing lookups. r=francois
MozReview-Commit-ID: 6kvM6z5OnPw
2017-01-26 11:36:52 +08:00
dimi
9cac186808 Bug 1328821 - hash completion request for v4 should not depend on table freshness. r=francois,henry
MozReview-Commit-ID: EIjDrnj1I4S
2017-01-17 08:33:08 +08:00
Henry Chang
b9d5f5080f Bug 1312339 - LookupResult to support variable length partial hash. r=francois
MozReview-Commit-ID: DKwNCNKJAW
2016-12-16 14:34:32 +08:00
Thomas Nguyen
7064eed520 Bug 1315386 - Make Safe Browsing code more shutdown-aware. r=francois,gcp.
MozReview-Commit-ID: ATCVfh5YLZl
2016-11-25 16:02:37 +08:00
Carsten "Tomcat" Book
2c983df934 Merge mozilla-central to autoland 2016-11-24 16:47:36 +01:00
Thomas Nguyen
6b8c52c162 Bug 1315893 - Add telemetry to measure update time for V2 and V4. r=francois
MozReview-Commit-ID: KtrVJXHXMo
2016-11-24 11:13:54 +08:00
Henry Chang
acfb7674fb Bug 1319286 - Cache nsIUrlClassifierDBService.getTables result until next update. r=francois.
MozReview-Commit-ID: ItjTQNzCVED
2016-11-22 10:39:58 +08:00
Iris Hsiao
5e3be007dc Backed out changeset b3574d03261b (bug 1319286) for GTest Assertion failures 2016-11-24 13:26:39 +08:00
Henry Chang
e0073e74a6 Bug 1319286 - Cache nsIUrlClassifierDBService.getTables result until next update. r=francois.
MozReview-Commit-ID: ItjTQNzCVED
2016-11-22 10:39:58 +08:00
Henry Chang
7f94b8915f Bug 1315097 - Build the provider dictionary on the main thread to be used everywhere. r=francois,gcp
MozReview-Commit-ID: Ft1deSNKuVB
2016-11-04 17:54:05 +08:00
Henry Chang
891cee9c33 Bug 1310142 - Preserve backup databases and raw table updates for diagnostic. r=francois
MozReview-Commit-ID: GeJoBrhuTgA
2016-10-14 17:58:18 +08:00
Thomas Nguyen
9a9556049a Bug 1298257 - Implement url matching for variable-length prefix set. r=dimi,gcp
MozReview-Commit-ID: 8Goh7yyAotN
2016-11-04 12:00:33 +08:00
dimi
6d436a40ee Bug 1305581 - Verify that V4 updates were applied correctly by computing a checksum on the final result. r=francois
MozReview-Commit-ID: LNtFOVMVw2U
2016-10-27 08:36:26 +08:00
DimiL
e841cd1751 Bug 1305780 - P1. Implement the update fail scheme for v4. r=gcp
MozReview-Commit-ID: LeVpVIUdmjc
2016-10-19 12:52:05 +08:00