Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Querying the macOS system quarantine database is relatively slow, so
we'd like to do it only once. This is relevant because the Telemetry
subsystem causes the attribution data to be fetched relatively early
during startup. By caching, we accept additional disk activity but
don't have to invoke an external process, query a database, etc.
We must make BROWSER_ATTRIBUTION_ERRORS histogram apply to macOS as
well as Windows. We add error codes to capture macOS-specific
detalis. And we push this out to a later cycle, since there's no
reason to revisit this immediately.
Differential Revision: https://phabricator.services.mozilla.com/D92695
On Windows, we parse the attribution data as if it was the
`URL.search` parameter. On macOS, we parse a URL slightly
differently: we parse it like `URL.searchParams` and we deduplicate
parameters.
For caching on macOS we need to serialize the parsed data. We could
use either approach but we elect to write the code serialized as it
would be on Windows, since that is the one that is getting actively
tested in the wild.
Differential Revision: https://phabricator.services.mozilla.com/D92694
The next step is to have the cached attribution file location vary
depending on the platform, and to have the tests run on additional
platforms.
Differential Revision: https://phabricator.services.mozilla.com/D84264
The issue this is addresses is that
[`CFURLCopyResourcePropertyForKey`](https://searchfox.org/mozilla-central/rev/ea7f70dac1c5fd18400f6d2a92679777d4b21492/xpcom/io/CocoaFileUtils.mm#212)
does not return quarantine data when launched as a GUI App. What
happens is that launching via the GUI requires the user to override
GateKeeper by going to Security & Privacy > Open Anyway. Doing that
updates the GateKeeper flags, and then the macOS API denies access:
once the GK flags reach some state, quarantine information is not
returned. This is not documented (as far as I can see) but moons ago,
[somebody else on the internet witnessed the same
thing](https://cocoa-dev.apple.narkive.com/kkYeAC8o/is-it-possible-to-read-your-own-quarantine-info-after-launch).
To work around, we run the system SQLite binary, to fish the relevant
information out of the per-user quarantine database. (SQLite is
installed by default on all relevant macOS versions.)
The most significant security concern I see is whether we can trust
this binary (in /usr/bin/sqlite3). Some discussion within the
Install/Update team suggested that an attacker who could corrupt or
modify that binary already had write access to the disk, which is an
attack vector equal to a totally compromised Firefox. If we determine
that we can't use the system SQLite binary, then we could use
Firefox's compiled copy of SQLite, but we might see versioning
issues. The system SQLite binary feels more robust.
This is implemented as a JS component for convenience, mostly: there
is no API for capturing output from `nsIProcess`. It would be
possible to maintain the existing XPCOM contract by renaming the
existing contract and adding a contract with a JS implementation that
passes through to the renamed implementation, but it doesn't seem
worth the effort.
In the next commits, we will generalize the existing caching mechanism
form Windows to also apply to macOS. This is mostly a performance
optimization, so that we sniff a single well-known location rather
than launching a process at each startup, although there is a
correctness argument here as well, since the quarantine database is
dynamic and the attribution URL could expire.
Differential Revision: https://phabricator.services.mozilla.com/D92693
The existing code handles Windows and macOS quite differently. On
macOS, the tests were not as comprehensive; this patch brings them
level and makes the form of the attribution data uniform on Windows
and macOS. In particular, attribution data fields will now be URI
coded, just as they are on Windows.
This will allow us to re-use the parsing machinery we have on Windows
when we cache attribution codes on macOS.
Differential Revision: https://phabricator.services.mozilla.com/D92692
A malformed URL returns null. The subsequent call to
`CocoaFileUtils::AddQuarantineMetadataToFile` does handle null
correctly, so this just avoids a crash.
Differential Revision: https://phabricator.services.mozilla.com/D92691
Querying the macOS system quarantine database is relatively slow, so
we'd like to do it only once. This is relevant because the Telemetry
subsystem causes the attribution data to be fetched relatively early
during startup. By caching, we accept additional disk activity but
don't have to invoke an external process, query a database, etc.
We must make BROWSER_ATTRIBUTION_ERRORS histogram apply to macOS as
well as Windows. We add error codes to capture macOS-specific
detalis. And we push this out to a later cycle, since there's no
reason to revisit this immediately.
Differential Revision: https://phabricator.services.mozilla.com/D92695
On Windows, we parse the attribution data as if it was the
`URL.search` parameter. On macOS, we parse a URL slightly
differently: we parse it like `URL.searchParams` and we deduplicate
parameters.
For caching on macOS we need to serialize the parsed data. We could
use either approach but we elect to write the code serialized as it
would be on Windows, since that is the one that is getting actively
tested in the wild.
Differential Revision: https://phabricator.services.mozilla.com/D92694
The next step is to have the cached attribution file location vary
depending on the platform, and to have the tests run on additional
platforms.
Differential Revision: https://phabricator.services.mozilla.com/D84264
The issue this is addresses is that
[`CFURLCopyResourcePropertyForKey`](https://searchfox.org/mozilla-central/rev/ea7f70dac1c5fd18400f6d2a92679777d4b21492/xpcom/io/CocoaFileUtils.mm#212)
does not return quarantine data when launched as a GUI App. What
happens is that launching via the GUI requires the user to override
GateKeeper by going to Security & Privacy > Open Anyway. Doing that
updates the GateKeeper flags, and then the macOS API denies access:
once the GK flags reach some state, quarantine information is not
returned. This is not documented (as far as I can see) but moons ago,
[somebody else on the internet witnessed the same
thing](https://cocoa-dev.apple.narkive.com/kkYeAC8o/is-it-possible-to-read-your-own-quarantine-info-after-launch).
To work around, we run the system SQLite binary, to fish the relevant
information out of the per-user quarantine database. (SQLite is
installed by default on all relevant macOS versions.)
The most significant security concern I see is whether we can trust
this binary (in /usr/bin/sqlite3). Some discussion within the
Install/Update team suggested that an attacker who could corrupt or
modify that binary already had write access to the disk, which is an
attack vector equal to a totally compromised Firefox. If we determine
that we can't use the system SQLite binary, then we could use
Firefox's compiled copy of SQLite, but we might see versioning
issues. The system SQLite binary feels more robust.
This is implemented as a JS component for convenience, mostly: there
is no API for capturing output from `nsIProcess`. It would be
possible to maintain the existing XPCOM contract by renaming the
existing contract and adding a contract with a JS implementation that
passes through to the renamed implementation, but it doesn't seem
worth the effort.
In the next commits, we will generalize the existing caching mechanism
form Windows to also apply to macOS. This is mostly a performance
optimization, so that we sniff a single well-known location rather
than launching a process at each startup, although there is a
correctness argument here as well, since the quarantine database is
dynamic and the attribution URL could expire.
Differential Revision: https://phabricator.services.mozilla.com/D92693
The existing code handles Windows and macOS quite differently. On
macOS, the tests were not as comprehensive; this patch brings them
level and makes the form of the attribution data uniform on Windows
and macOS. In particular, attribution data fields will now be URI
coded, just as they are on Windows.
This will allow us to re-use the parsing machinery we have on Windows
when we cache attribution codes on macOS.
Differential Revision: https://phabricator.services.mozilla.com/D92692
A malformed URL returns null. The subsequent call to
`CocoaFileUtils::AddQuarantineMetadataToFile` does handle null
correctly, so this just avoids a crash.
Differential Revision: https://phabricator.services.mozilla.com/D92691
Querying the macOS system quarantine database is relatively slow, so
we'd like to do it only once. This is relevant because the Telemetry
subsystem causes the attribution data to be fetched relatively early
during startup. By caching, we accept additional disk activity but
don't have to invoke an external process, query a database, etc.
We must make BROWSER_ATTRIBUTION_ERRORS histogram apply to macOS as
well as Windows. We add error codes to capture macOS-specific
detalis. And we push this out to a later cycle, since there's no
reason to revisit this immediately.
Differential Revision: https://phabricator.services.mozilla.com/D92695
On Windows, we parse the attribution data as if it was the
`URL.search` parameter. On macOS, we parse a URL slightly
differently: we parse it like `URL.searchParams` and we deduplicate
parameters.
For caching on macOS we need to serialize the parsed data. We could
use either approach but we elect to write the code serialized as it
would be on Windows, since that is the one that is getting actively
tested in the wild.
Differential Revision: https://phabricator.services.mozilla.com/D92694
The next step is to have the cached attribution file location vary
depending on the platform, and to have the tests run on additional
platforms.
Differential Revision: https://phabricator.services.mozilla.com/D84264
The issue this is addresses is that
[`CFURLCopyResourcePropertyForKey`](https://searchfox.org/mozilla-central/rev/ea7f70dac1c5fd18400f6d2a92679777d4b21492/xpcom/io/CocoaFileUtils.mm#212)
does not return quarantine data when launched as a GUI App. What
happens is that launching via the GUI requires the user to override
GateKeeper by going to Security & Privacy > Open Anyway. Doing that
updates the GateKeeper flags, and then the macOS API denies access:
once the GK flags reach some state, quarantine information is not
returned. This is not documented (as far as I can see) but moons ago,
[somebody else on the internet witnessed the same
thing](https://cocoa-dev.apple.narkive.com/kkYeAC8o/is-it-possible-to-read-your-own-quarantine-info-after-launch).
To work around, we run the system SQLite binary, to fish the relevant
information out of the per-user quarantine database. (SQLite is
installed by default on all relevant macOS versions.)
The most significant security concern I see is whether we can trust
this binary (in /usr/bin/sqlite3). Some discussion within the
Install/Update team suggested that an attacker who could corrupt or
modify that binary already had write access to the disk, which is an
attack vector equal to a totally compromised Firefox. If we determine
that we can't use the system SQLite binary, then we could use
Firefox's compiled copy of SQLite, but we might see versioning
issues. The system SQLite binary feels more robust.
This is implemented as a JS component for convenience, mostly: there
is no API for capturing output from `nsIProcess`. It would be
possible to maintain the existing XPCOM contract by renaming the
existing contract and adding a contract with a JS implementation that
passes through to the renamed implementation, but it doesn't seem
worth the effort.
In the next commits, we will generalize the existing caching mechanism
form Windows to also apply to macOS. This is mostly a performance
optimization, so that we sniff a single well-known location rather
than launching a process at each startup, although there is a
correctness argument here as well, since the quarantine database is
dynamic and the attribution URL could expire.
Differential Revision: https://phabricator.services.mozilla.com/D92693
The existing code handles Windows and macOS quite differently. On
macOS, the tests were not as comprehensive; this patch brings them
level and makes the form of the attribution data uniform on Windows
and macOS. In particular, attribution data fields will now be URI
coded, just as they are on Windows.
This will allow us to re-use the parsing machinery we have on Windows
when we cache attribution codes on macOS.
Differential Revision: https://phabricator.services.mozilla.com/D92692
A malformed URL returns null. The subsequent call to
`CocoaFileUtils::AddQuarantineMetadataToFile` does handle null
correctly, so this just avoids a crash.
Differential Revision: https://phabricator.services.mozilla.com/D92691
Querying the macOS system quarantine database is relatively slow, so
we'd like to do it only once. This is relevant because the Telemetry
subsystem causes the attribution data to be fetched relatively early
during startup. By caching, we accept additional disk activity but
don't have to invoke an external process, query a database, etc.
We must make BROWSER_ATTRIBUTION_ERRORS histogram apply to macOS as
well as Windows. We add error codes to capture macOS-specific
detalis. And we push this out to a later cycle, since there's no
reason to revisit this immediately.
Differential Revision: https://phabricator.services.mozilla.com/D92695
On Windows, we parse the attribution data as if it was the
`URL.search` parameter. On macOS, we parse a URL slightly
differently: we parse it like `URL.searchParams` and we deduplicate
parameters.
For caching on macOS we need to serialize the parsed data. We could
use either approach but we elect to write the code serialized as it
would be on Windows, since that is the one that is getting actively
tested in the wild.
Differential Revision: https://phabricator.services.mozilla.com/D92694
The next step is to have the cached attribution file location vary
depending on the platform, and to have the tests run on additional
platforms.
Differential Revision: https://phabricator.services.mozilla.com/D84264
The issue this is addresses is that
[`CFURLCopyResourcePropertyForKey`](https://searchfox.org/mozilla-central/rev/ea7f70dac1c5fd18400f6d2a92679777d4b21492/xpcom/io/CocoaFileUtils.mm#212)
does not return quarantine data when launched as a GUI App. What
happens is that launching via the GUI requires the user to override
GateKeeper by going to Security & Privacy > Open Anyway. Doing that
updates the GateKeeper flags, and then the macOS API denies access:
once the GK flags reach some state, quarantine information is not
returned. This is not documented (as far as I can see) but moons ago,
[somebody else on the internet witnessed the same
thing](https://cocoa-dev.apple.narkive.com/kkYeAC8o/is-it-possible-to-read-your-own-quarantine-info-after-launch).
To work around, we run the system SQLite binary, to fish the relevant
information out of the per-user quarantine database. (SQLite is
installed by default on all relevant macOS versions.)
The most significant security concern I see is whether we can trust
this binary (in /usr/bin/sqlite3). Some discussion within the
Install/Update team suggested that an attacker who could corrupt or
modify that binary already had write access to the disk, which is an
attack vector equal to a totally compromised Firefox. If we determine
that we can't use the system SQLite binary, then we could use
Firefox's compiled copy of SQLite, but we might see versioning
issues. The system SQLite binary feels more robust.
This is implemented as a JS component for convenience, mostly: there
is no API for capturing output from `nsIProcess`. It would be
possible to maintain the existing XPCOM contract by renaming the
existing contract and adding a contract with a JS implementation that
passes through to the renamed implementation, but it doesn't seem
worth the effort.
In the next commits, we will generalize the existing caching mechanism
form Windows to also apply to macOS. This is mostly a performance
optimization, so that we sniff a single well-known location rather
than launching a process at each startup, although there is a
correctness argument here as well, since the quarantine database is
dynamic and the attribution URL could expire.
Differential Revision: https://phabricator.services.mozilla.com/D92693
The existing code handles Windows and macOS quite differently. On
macOS, the tests were not as comprehensive; this patch brings them
level and makes the form of the attribution data uniform on Windows
and macOS. In particular, attribution data fields will now be URI
coded, just as they are on Windows.
This will allow us to re-use the parsing machinery we have on Windows
when we cache attribution codes on macOS.
Differential Revision: https://phabricator.services.mozilla.com/D92692
A malformed URL returns null. The subsequent call to
`CocoaFileUtils::AddQuarantineMetadataToFile` does handle null
correctly, so this just avoids a crash.
Differential Revision: https://phabricator.services.mozilla.com/D92691
