Nullable<PersistenceType> only allows to specify a concrete persistence type or
all persistence types while PersistenceScope is intended to allow specification
of multiple persistence types as well. Support for that will be added in a
separate patch. This patch is about using the new type especially in directory
locks.
Differential Revision: https://phabricator.services.mozilla.com/D195373
Cached origin usage can be currently obtained by passing fromMemory=true to
nsIQuotaManagerService::GetUsageForPrincipal. However, supporting both ways of
getting origin on one operation is not ideal for sevaral reasons. The return
value is a plain integer in the case of cached origin usage and getting cached
origin usage in theory doesn't have to go the QM IO thread. So it would be
better to have a dedicated method for that called
nsIQuotaManagerService::GetCachedUsageForPrincipal.
Differential Revision: https://phabricator.services.mozilla.com/D195360
Currently, getting cached origin usage may trigger temporary storage
initialization which is a problem in tests when we want just compare real and
cached usage without affecting the state of initialization.
This patch changes the behavior to return Nothing if temporary storage is not
initialized.
Consumers may either explicitly make sure that temporary storage is initialized
before getting cached origin usage or they can use the other mode when real
usage is colleced from disk.
Differential Revision: https://phabricator.services.mozilla.com/D194504
Sub actors are still created, but their only purpose is to allow cancellation
of alreaady created requests. Actual results are now returned as asynchronous
responses to the asynchronous messages.
Differential Revision: https://phabricator.services.mozilla.com/D194097
The class which implements the protocol is already refcounted, but the
interaction with IPC still uses manual deallocation.
Differential Revision: https://phabricator.services.mozilla.com/D194042
This is one of the things which are needed for making GetUsageOp and
GetOriginUsageOp independent from the actor. In practise, for changing the main
base class of the operations from QuotaUsageRequestBase to
ResolvableNormalOriginOp.
Differential Revision: https://phabricator.services.mozilla.com/D194011
There are currently EnsurePersistentOriginIsInitializedInternal and
EnsureTemporaryOriginIsInitializedInternal methods which can only be
called on the QuotaManager IO thread. These methods shouldn't be exposed to
quota clients and origin operations. There should be public methods callable
from the PBackground thread returning a MozPromise instead. Such methods will
guarantee that proper directory locking is acquired before persistent origin
initialization or temporary origin initialization is started.
Differential Revision: https://phabricator.services.mozilla.com/D194005
One of the goals of the asynchronous temporary storage initialization is to
call Ensure(Persistent|Temporary)OriginIsInitialized only from
Initialize(Persistent|Temporary)OriginOp. Calling from other places including
quota clients will be disallowed by changing the method to a private method.
The private nature of the method should be emphasized by adding the Internal
suffix.
Differential Revision: https://phabricator.services.mozilla.com/D192150
Counting of clear/shutdown storage operations was only a temporary solution.
We are now approaching a point when we will be able to initialize origins
asynchronously and that can't efficiently work with counting of clear origin
operations because initialization and clearing of origins is parameterized by
persistence type and actual origin. So there can be a clear origin operation
which doesn't block other init origin operation. Couting of clear operations
would be problematic in that case.
Evaluation of existing directory locks should work both for storage and origin
initialization.
Differential Revision: https://phabricator.services.mozilla.com/D193191
Until now, directory locks were dropped when the last strong reference was
removed or after calling Drop explicitly. The dependency on ref-counting makes
it less obvious when directory locks are dropped for real and it's also
difficult to release them asynchronously eventually. This patch removes the
directory lock unregistration from the destructor, so from now on, directory
locks must be always dropped explicitly.
Differential Revision: https://phabricator.services.mozilla.com/D197294
There's currently EnsureTemporaryStorageIsInitializedInternal which can only be
called on the QuotaManager IO thread. That method shouldn't be exposed to quota
clients and origin operations. There should be a public method callable from
the PBackground thread returning a MozPromise instead. Such method will
guarantee that proper directory locking is acquired before temporary storage
initialization is started.
Differential Revision: https://phabricator.services.mozilla.com/D192135
One of the goals of the asynchronous temporary storage initialization is to
call EnsureTemporaryStorageIsInitialized only from InitTemporaryStorageOp.
Calling from other places including quota clients will be disallowed by
changing the method to a private method. The private nature of the method
should be emphasized by adding the Internal suffix.
Differential Revision: https://phabricator.services.mozilla.com/D188332
This patch also adds QuotaManager::StorageInitialized and
QuotaManager::TemporaryStorageInitialized which can be then used by the
QuotaManagerDependencyFixture.
Differential Revision: https://phabricator.services.mozilla.com/D191930
Operations are shown as "pending" in the QM shutdown hang annotations until they are fully executed. Given that executing them often requires acquiring a directory lock first and also involves some thread hopping, we want to understand better, where we hang.
In particular many crashes show `ShutdownStorageOp` as pending but there is no sign in the stack trace that `DoDirectoryWork` ever started. In addition, most if not all of those crashes show either an open IDB connection or an unclosed Cache manager, which might interfere with directory locking here.
The annotation should help to confirm that `ShutdownStorageOp` is endlessly waiting to acquire the directory lock in those cases.
Differential Revision: https://phabricator.services.mozilla.com/D190425
ClearRequestBase::DeleteFiles currently provides a way to clear origins for any
combination of persistence type, origin scope and client type. All origin
directories need to be traversed to find relevant matches. This can be slow if
there are many origin directories. Fortunately, the traversal can be completely
avoided when exact origin is being cleared.
Changes done in this patch:
- added a dedicated ClearRequestBase::DeleteFiles method for clearing of exact
origins
- changed ClearOriginOp::DoDirectoryWork to use the new variant of
ClearRequestBase::DeleteFiles
Differential Revision: https://phabricator.services.mozilla.com/D186781
The removing of an origin directory became a bit more complex because when the
app shutdown already started, origin directories shouldn't be fully removed
from disk. They need to be only moved to a special directory. All this
complexity should be covered by a dedicated QuotaManager method.
Changes done in this patch:
- added a new method QuotaManager::RemoveOriginDirectory
- added a prefilled string for the special to-be-removed directory
- adjusted ClearRequestBase::DeleteFiles to use the new
QuotaManager::RemoveOriginDirectory method
Differential Revision: https://phabricator.services.mozilla.com/D186780
There are now no callers of nsIQuotaManagerService::ClearStoragesForPrincipal
which would request clearing of all storages for given prefix. The support for
that can be removed.
Changed done in this patch:
- removed the aClearAll argument from
nsIQuotaManagerService::ClearStoragesForPrincipal
- removed the aClearAll argument from the async IPC message
- removed the aClearAll argument from QuotaManager::ClearStoragesForOrigin
- changed ClearOriginOp to support clearing of exact origins only
Differential Revision: https://phabricator.services.mozilla.com/D186779
nsIQuotaManagerService::ClearStoragesForPrincipal currently supports both
clearing storages for a specific origin only and clearing storages for a group
of origins sharing the same prefix. It would be better to have separate methods
for this.
Changes done in this patch:
- added nsIQuotaManagerService::ClearStoragesForOriginPrefix
- added a new helper for testing clearOriginsByPrefix
- changed verifyStorage to accept an optional shared key name
- added thorough testing for the new method
Differential Revision: https://phabricator.services.mozilla.com/D186777
QuotaManagerService::ClearStoragesForOriginAttributesPattern and
QuotaManagerService::ClearStoragesForPrincipal still create sub actors which
makes it hard to add new clearing operations which would use async IPC messages
and which would inherit from the ClearRequestBase class as well.
Changes done in this patch:
- added QuotaManager::ClearStoragesForOrigin
- added QuotaManager::ClearStoragesForOriginAttributesPattern
- changed ClearRequestBase to inherit from ResolvableNormalOriginOp
- QuotaManagerService::ClearStoragesForOriginAttributesPattern reworked to use
an asynchronous message instead of a sub actor
- QuotaManagerService::ClearStoragesForPrincipal reworked to use an
asynchronous message instead of a sub actor
- added a new mactor QM_CUF_AND_IPC_FAIL similar to QM_IPC_FAIL
creating a sub actor
Differential Revision: https://phabricator.services.mozilla.com/D186628
ClearRequestBase currently provides generic directory locking based on generic
member variables. It's not totally clear what derived classes do in terms of
directory locking and work on the QuotaManager IO thread. It would be better if
ClearRequestBase only provided a generic DeleteFiles function instead.
Changed done in this patch:
- moved implementation of directory locking methods to derived classes of
ClearRequestBase
- moved implementation of DoDirectoryWork to derived classes as well
- adjusted member variables
Differential Revision: https://phabricator.services.mozilla.com/D186627
Origin operations which require storage initialization currently create a
universal directory lock first and then when the universal directorylock is
acquired, they bounce to the QuotaManager I/O thread where they ensure that
storage is initialized. This can be now replaced by just calling
QuotaManager::OpenStorageDirectory.
Changes done in this patch:
- replaced QuotaManager::CreateDirectoryLockInternal call with
QuotaManager::OpenStorageDirectory in corresponding OpenDirectory
implementations
- removed QuotaManager::EnsureStorageIsInitializedInternal call from
corresponding DoDirectoryWork implementations
- added QuotaManager::AssertStorageIsInitializedInternal to corresponding
DoDirectoryWork implementations
Differential Revision: https://phabricator.services.mozilla.com/D186206
SaveOriginAccessTimeOp::DoDirectoryWork currently doesn't call
QuotaManager::EnsureStorageIsInitializedInternal and just expects that
something else initialized storage previously. This seems to work, but it would
be cleaner to always make sure that storage is initialized. However, adding
QuotaManager::EnsureStorageIsInitializedInternal revealed another problem.
Storage shudown or storage clearing acquires an exlusive lock over entire
storage area which essentially forces that all existing directory locks are
released first. When the last directory lock for an origin is released, saving
of origin access time is scheduled. The problem is that it's scheduled after
the exclusive lock for storage shutdown or storage clearing, so storage would
be initialized again in the end or access time wouldn't be saved at all due
to quota manager shutdown being already in progress.
Changes done in this patch:
- added QuotaManager::EnsureStorageIsInitializedInternal call to
SaveOriginAccessTimeOp::DoDirectoryWork
- changed QuotaManager::UnregisterDirectoryLock to work with already cleared
directory lock tables
- added a new QuotaManager::ClearDirectoryLockTables method
- added QuotaManager::ClearDirectoryLockTables call to
ShutdownStorageOp::OpenDirectory and ClearStorageOp::OpenDirectory
- made ClearStorageOp and ShutdownStorageOp friend classes of QuotaManager
Differential Revision: https://phabricator.services.mozilla.com/D187877
Some origin operations are currently not protected by directory locks because
they use cached data for generating responses. However these origin operations
ensure that storage is initializated on QuotaManager IO thread prior using the
cache data, so they should have directory directory locks as well.
Changed done in this patch:
- added directory locks to origin operations which require storage
initialization
- changed corresponding OpenDirectory implementations
- added corresponding CloseDirectory implementations
Differential Revision: https://phabricator.services.mozilla.com/D186205
LoadArchivedOrigins (called from QuotaClient::AboutToClearOrigins) currently
ensures that storage is initialized which eventually creates the archive from
webappsstore.sqlite. Conceptually, storage should be always initialized before
QuotaClient::AboutToClearOrigins is called.
Changes done in this patch:
- removed QuotaManager::EnsureStorageIsInitializedInternal call from
LoadArchivedOrigins
- added QuotaManger::EnsureStorageIsInitializedInternal call to
ClearStorageOp::DoDirectoryWork
Differential Revision: https://phabricator.services.mozilla.com/D186115
Now when origin operations have been refactored to do directory locking on
their own, it's possibly to pass an already acquired directory lock to the
InitOp object. This is required to fix a flaw in the implementation of
QuotaManager::OpenClientDirectory. Basically a directory lock for the InitOp
and a client directory lock both need to be acquired at the same time to
maintain correct ordering of operations.
Changes done in this patch:
- added a new overload for QuotaManager::InitializeStorage which takes a
directory lock
- changed InitOp to work with an already acquired directory lock
- enabled a disabled test for QuotaManager::OpenClientDirectory
- added more tests for QuotaManager::OpenClientDirectory
Differential Revision: https://phabricator.services.mozilla.com/D186080
Some operations will need to do more complex directory opening involving for
example acquiring multiple directory locks. For this reason it's no longer
desired to hold a directory lock in the NormalOriginOperationBase class.
Changed done in this patch:
- changed NormalOriginOperationsBase::CreateDirectoryLock too return a
MozPromise
- renamed NormalOriginOperationsBase::CreatedDirectoryLock to OpenDirectory
- added a new pure virtual method NormalOriginOperationBase::CloseDirectory
- removed mDirectoryLock member from NormalOriginOperationBase
- adjusted derived classes of NormalOriginOperationsBase
Differential Revision: https://phabricator.services.mozilla.com/D186079
Now when NormalOriginOperationBase lets the derived classes to create directory
locks, it is no longer needed to keep members related to directory locking in
the NormalOriginOperationBase class.
Changes done in this patch:
- optimized members of derived classes
- adjusted CreateDirectoryLock implementations
- removed members related to directory locking from NormalOriginOperationBase
- adjusted constructors of derived classes
Differential Revision: https://phabricator.services.mozilla.com/D185647
Directory locks are currently created either by NormalOriginOperationBase
based on the parameters passed to its constructor or derived classes override
the CreateDirectoryLock method. Some operations will need to acquire multiple
locks in future and eventually do some other stuff before they can start doing
IO work, so it would be better to remove the default implementation of
CreateDirectoryLock in advance and let the derived classes to always implement
the method.
Changes done in this patch:
- copied the default implementation of CreateDirectoryLock to each origin
operation
- changed NormalOriginOperationBase::CreateDirectoryLock to be a pure virtual
method
- removed the default implementation of CreateDirectoryLock
- added some missing AssertIsOnOwningThread calls
Differential Revision: https://phabricator.services.mozilla.com/D185646
OriginOperationBase and the derived classes currently must get QuotaManager by
calling QuotaManager::Get() and usually asserting that the returned value is
not null. Now when OriginOperationBase is created and destroyed only on the
PBackground thread, a new not null strong reference to QuotaManager can be
added to OriginOperationBase which will eliminate the need for calling
QuotaManager::Get. Calling QuotaManager::Get less is desired because such a
call can get more expensive in future.
Changes done in this patch:
- added not null strong reference to QuotaManager from OriginOperationBase
- adjusted constructors of derived classes
- adjusted factories for origin operations
- started using the new member in some places (instead of calling QuotaManager::Get)
- removed some now unnecessary QuotaManager::Get calls
Differential Revision: https://phabricator.services.mozilla.com/D185645
OriginOperationBase currently uses manual dispatching of runnables controlled
by a state machine which is not that easy to understand and requires many
helper methods. Additionally, OriginOperationBase can be destroyed off the
PBackground thread which makes it hard to hold PBackground only objects.
Changes done in this patch:
- changed OriginOperationBase to not inherit from Runnable
- changed OriginOperationBase to be created and destroyed on the PBackground thread only
- replaced the manual dispatching of runnables with a MozPromise chain
- removed OriginOperationBase::Dispatch
- changed pure virtual method OriginOperationBase::Open to return a MozPromise
- changed FinalizeOriginEvictionOp to be always created on the PBackground
thread only (as a consequence of similar changes done to OriginOperationBase)
Differential Revision: https://phabricator.services.mozilla.com/D185641
There's currently EnsureStorageIsInitializedInternal which can only be called
on the QuotaManager IO thread. That method shouldn't be exposed to quota
clients and origin operations. There should be a public method callable from
the PBackground thread returning a MozPromise instead. Such method will
guarantee that proper directory locking is acquired before storage
initialization is started.
Changes done in this patch:
- added QuotaManager::InitializeStorage method
- added QuotaManager::IsStorageInitialized method
- QuotaManagerService::Init reworked to use an asynchronous message instead of
creating a sub actor
- added QuotaManagerDependencyFixture::IsStorageInitialized
- added QuotaManagerDependencyFixture::AssertStorageIsInitialized
- added QuotaManagerDependencyFixture::AssertStorageIsNotInitialized
- eliminated QuotaManager::IsStorageInitializedInternal calls in tests
- eliminated QuotaManager::EnsureStorageIsInitializedInternal calls in tests
- added a bunch of tests for the new QuotaManager::InitializeStorage method
Differential Revision: https://phabricator.services.mozilla.com/D185570
One of the goals of bug 1749504 is to call EnsureStorageIsInitialized only from
InitOp. Calling from other places including quota clients will be disallowed
by changing the method to a private method. The private nature of the method
should be emphasized by adding the Internal suffix.
Changes done in this patch:
- IsStorageInitialized renamed to IsStorageInitializedInternal
- AssertStorageIsInitialized renamed to AssertStorageIsInitializedInternal
- EnsureStorageIsInitialized renamed to EnsureStorageIsInitializedInternal
Differential Revision: https://phabricator.services.mozilla.com/D185547
There's currently no QuotaManager method for triggering storage clearing which
could be used by the ClearOp or directly in gtests. Asynchronous storage
initialization will introduce a flag which needs to be unset after storage
clearing or storage resetting. This can't be easilly done without a
QuotaManager method for storage clearing.
Changes done in this patch:
- added QuotaManager::ClearStorage method
- renamed ClearOp to ClearStorageOp
- QuotaManagerService::Clear reworked to use an asynchronous message instead of
creating a sub actor
Differential Revision: https://phabricator.services.mozilla.com/D185546
The ResetOrClearOp currently supports both the resetting and clearing of
storage. However, there's already an operation for resetting storage called
ShutdownStorageOp which can be used separately.
Changes done in this patch:
- ResetOrClearOp renamed to ClearOp
- ClearOp converted to support only storage clearing
- QuotaManagerService::Reset reworked to use an asynchronous message instead of
creating a sub actor
- added generic helpers/callbacks for handling returned MozPromises
Differential Revision: https://phabricator.services.mozilla.com/D185545
