Bug 1413841 - Check for integer overflow in AesTask::DoCrypto() r=keeler

Summary:
After calling mResult.SetLength(mData.Length() + 16) we should check that the
integer addition didn't overflow. It seems at the moment impossible to create
ArrayBuffers of size >= 0x0xfffffff0, however adding a check here doesn't hurt.

mResult.Length() is passed to the PK11 API functions as a maxOut parameter and
/should/ be checked by the softoken crypto algorithm implementations. AES-ECB
and AES-GCM seem to do that correctly.

Reviewers: keeler

Reviewed By: keeler

Subscribers: mcote, ttaubert, jcj, keeler

Bug #: 1413841

Differential Revision: https://phabricator.services.mozilla.com/D188

dom/crypto/WebCryptoTask.cpp

@@ -717,12 +717,16 @@ private:
       return NS_ERROR_DOM_INVALID_ACCESS_ERR;
     }
 
+    // Check whether the integer addition would overflow.
+    if (std::numeric_limits<CryptoBuffer::size_type>::max() - 16 < mData.Length()) {
+      return NS_ERROR_DOM_DATA_ERR;
+    }
+
     // Initialize the output buffer (enough space for padding / a full tag)
-    uint32_t dataLen = mData.Length();
-    uint32_t maxLen = dataLen + 16;
-    if (!mResult.SetLength(maxLen, fallible)) {
+    if (!mResult.SetLength(mData.Length() + 16, fallible)) {
       return NS_ERROR_DOM_UNKNOWN_ERR;
     }
+
     uint32_t outLen = 0;
 
     // Perform the encryption/decryption