======================================
Client Side Operations Timeouts Tests
======================================

.. contents::

----

Introduction
============

This document describes the tests that drivers MUST run to validate the behavior of the timeoutMS option. These tests
are broken up into automated YAML/JSON tests and additional prose tests.

Spec Tests
==========

This directory contains a set of YAML and JSON spec tests. Drivers MUST run these as described in the "Unified Test
Runner" specification. Because the tests introduced in this specification are timing-based, there is a risk that some
of them may intermittently fail without any bugs being present in the driver. As a mitigatio, drivers MAY execute
these tests in two new Evergreen tasks that use single-node replica sets: one with only authentication enabled and
another with both authentication and TLS enabled. Drivers that choose to do so SHOULD use the ``single-node-auth.json``
and ``single-node-auth-ssl.json`` files in the ``drivers-evergreen-tools`` repository to create these clusters.

Prose Tests
===========

There are some tests that cannot be expressed in the unified YAML/JSON format. For each of these tests, drivers MUST
create a MongoClient without the ``timeoutMS`` option set (referred to as ``internalClient``). Any fail points set
during a test MUST be unset using ``internalClient`` after the test has been executed. All MongoClient instances
created for tests MUST be configured with read/write concern ``majority``, read preference ``primary``, and command
monitoring enabled to listen for ``command_started`` events.

Multi-batch writes
~~~~~~~~~~~~~~~~~~

This test MUST only run against server versions 4.4 and higher.

#. Using ``internalClient``, drop the ``db.coll`` collection.
#. Using ``internalClient``, set the following fail point:

   .. code:: javascript

       {
           configureFailPoint: "failCommand",
           mode: {
               times: 2
           },
           data: {
               failCommands: ["insert"],
               blockConnection: true,
               blockTimeMS: 15
           }
       }

#. Create a new MongoClient (referred to as ``client``) with ``timeoutMS=20``.
#. Using ``client``, insert 100,001 empty documents in a single ``insertMany`` call.

   - Expect this to fail with a timeout error.

#. Verify that two ``insert`` commands were executed against ``db.coll`` as part of the ``insertMany`` call.

maxTimeMS is not set for commands sent to mongocryptd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This test MUST only be run against enterprise server versions 4.2 and higher.

#. Launch a mongoryptd process on 23000.
#. Create a MongoClient (referred to as ``client``) using the URI ``mongodb://localhost:23000/?timeoutMS=1000``.
#. Using ``client``, execute the ``{ ping: 1 }`` command against the ``admin`` database.
#. Verify via command monitoring that the ``ping`` command sent did not contain a ``maxTimeMS`` field.

ClientEncryption
~~~~~~~~~~~~~~~~

Each test under this category MUST only be run against server versions 4.4 and higher. In these tests,
``LOCAL_MASTERKEY`` refers to the following base64:

.. code:: javascript

  Mng0NCt4ZHVUYUJCa1kxNkVyNUR1QURhZ2h2UzR2d2RrZzh0cFBwM3R6NmdWMDFBMUN3YkQ5aXRRMkhGRGdQV09wOGVNYUMxT2k3NjZKelhaQmRCZGJkTXVyZG9uSjFk

For each test, perform the following setup:

#. Using ``internalClient``, drop and create the ``keyvault.datakeys`` collection.
#. Create a MongoClient (referred to as ``keyVaultClient``) with ``timeoutMS=10``.
#. Create a ``ClientEncryption`` object that wraps ``keyVaultClient`` (referred to as ``clientEncryption``). Configure this object with ``keyVaultNamespace`` set to ``keyvault.datakeys`` and the following KMS providers map:

   .. code:: javascript

       {
           "local": { "key": <base64 decoding of LOCAL_MASTERKEY> }
       }

createDataKey
`````````````

#. Using ``internalClient``, set the following fail point:

   .. code:: javascript

       {
           configureFailPoint: "failCommand",
           mode: {
               times: 1
           },
           data: {
               failCommands: ["insert"],
               blockConnection: true,
               blockTimeMS: 15
           }
       }

#. Call ``clientEncryption.createDataKey()`` with the ``local`` KMS provider.

   - Expect this to fail with a timeout error.

#. Verify that an ``insert`` command was executed against to ``keyvault.datakeys`` as part of the ``createDataKey`` call.

encrypt
```````

#. Call ``client_encryption.createDataKey()`` with the ``local`` KMS provider.

   - Expect a BSON binary with subtype 4 to be returned, referred to as ``datakeyId``.

#. Using ``internalClient``, set the following fail point:

   .. code:: javascript

       {
           configureFailPoint: "failCommand",
           mode: {
               times: 1
           },
           data: {
               failCommands: ["find"],
               blockConnection: true,
               blockTimeMS: 15
           }
       }

#. Call ``clientEncryption.encrypt()`` with the value ``hello``, the algorithm ``AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic``, and the keyId ``datakeyId``.

   - Expect this to fail with a timeout error.

#. Verify that a ``find`` command was executed against the ``keyvault.datakeys`` collection as part of the ``encrypt`` call.

decrypt
```````

#. Call ``clientEncryption.createDataKey()`` with the ``local`` KMS provider.

   - Expect this to return a BSON binary with subtype 4, referred to as ``dataKeyId``.

#. Call ``clientEncryption.encrypt()`` with the value ``hello``, the algorithm ``AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic``, and the keyId ``dataKeyId``.

   - Expect this to return a BSON binary with subtype 6, referred to as ``encrypted``.

#. Close and re-create the ``keyVaultClient`` and ``clientEncryption`` objects.

#. Using ``internalClient``, set the following fail point:

   .. code:: javascript

       {
           configureFailPoint: "failCommand",
           mode: {
               times: 1
           },
           data: {
               failCommands: ["find"],
               blockConnection: true,
               blockTimeMS: 15
           }
       }

#. Call ``clientEncryption.decrypt()`` with the value ``encrypted``.

   - Expect this to fail with a timeout error.

#. Verify that a ``find`` command was executed against the ``keyvault.datakeys`` collection as part of the ``decrypt`` call.

Background Connection Pooling
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The tests in this section MUST only be run if the server version is 4.4 or higher and the URI has authentication
fields (i.e. a username and password). Each test in this section requires drivers to create a MongoClient and then wait
for some CMAP events to be published. Drivers MUST wait for up to 10 seconds and fail the test if the specified events
are not published within that time.

timeoutMS used for handshake commands
`````````````````````````````````````

#. Using ``internalClient``, set the following fail point:

   .. code:: javascript

       {
           configureFailPoint: "failCommand",
           mode: {
               times: 1
           },
           data: {
               failCommands: ["saslContinue"],
               blockConnection: true,
               blockTimeMS: 15,
               appName: "timeoutBackgroundPoolTest"
           }
       }

#. Create a MongoClient (referred to as ``client``) configured with the following:

   - ``minPoolSize`` of 1
   - ``timeoutMS`` of 10
   - ``appName`` of ``timeoutBackgroundPoolTest``
   - CMAP monitor configured to listen for ``ConnectionCreatedEvent`` and ``ConnectionClosedEvent`` events.

#. Wait for a ``ConnectionCreatedEvent`` and a ``ConnectionClosedEvent`` to be published.

timeoutMS is refreshed for each handshake command
`````````````````````````````````````````````````

#. Using ``internalClient``, set the following fail point:

   .. code:: javascript

       {
           configureFailPoint: "failCommand",
           mode: "alwaysOn",
           data: {
               failCommands: ["isMaster", "saslContinue"],
               blockConnection: true,
               blockTimeMS: 15,
               appName: "refreshTimeoutBackgroundPoolTest"
           }
       }

#. Create a MongoClient (referred to as ``client``) configured with the following:

   - ``minPoolSize`` of 1
   - ``timeoutMS`` of 20
   - ``appName`` of ``refreshTimeoutBackgroundPoolTest``
   - CMAP monitor configured to listen for ``ConnectionCreatedEvent`` and ``ConnectionReady`` events.

#. Wait for a ``ConnectionCreatedEvent`` and a ``ConnectionReady`` to be published.

Blocking Iteration Methods
~~~~~~~~~~~~~~~~~~~~~~~~~~

Tests in this section MUST only be run against server versions 4.4 and higher and only apply to drivers that have a
blocking method for cursor iteration that executes ``getMore`` commands in a loop until a document is available or an
error occurs.

Tailable cursors
````````````````

#. Using ``internalClient``, drop the ``db.coll`` collection.
#. Using ``internalClient``, insert the document ``{ x: 1 }`` into ``db.coll``.
#. Using ``internalClient``, set the following fail point:

   .. code:: javascript

       {
           configureFailPoint: "failCommand",
           mode: "alwaysOn",
           data: {
               failCommands: ["getMore"],
               blockConnection: true,
               blockTimeMS: 15
           }
       }

#. Create a new MongoClient (referred to as ``client``) with ``timeoutMS=20``.
#. Using ``client``, create a tailable cursor on ``db.coll`` with ``cursorType=tailable``.

   - Expect this to succeed and return a cursor with a non-zero ID.

#. Call either a blocking or non-blocking iteration method on the cursor.

   - Expect this to succeed and return the document ``{ x: 1 }`` without sending a ``getMore`` command.

#. Call the blocking iteration method on the resulting cursor.

   - Expect this to fail with a timeout error.

#. Verify that a ``find`` command and two ``getMore`` commands were executed against the ``db.coll`` collection during the test.

Change Streams
``````````````

#. Using ``internalClient``, drop the ``db.coll`` collection.
#. Using ``internalClient``, set the following fail point:

   .. code:: javascript

       {
           configureFailPoint: "failCommand",
           mode: "alwaysOn",
           data: {
               failCommands: ["getMore"],
               blockConnection: true,
               blockTimeMS: 15
           }
       }

#. Create a new MongoClient (referred to as ``client``) with ``timeoutMS=20``.
#. Using ``client``, use the ``watch`` helper to create a change stream against ``db.coll``.

   - Expect this to succeed and return a change stream with a non-zero ID.

#. Call the blocking iteration method on the resulting change stream.

   - Expect this to fail with a timeout error.

#. Verify that an ``aggregate`` command and two ``getMore`` commands were executed against the ``db.coll`` collection during the test.

GridFS - Upload
~~~~~~~~~~~~~~~

Tests in this section MUST only be run against server versions 4.4 and higher.

uploads via openUploadStream can be timed out
`````````````````````````````````````````````

#. Using ``internalClient``, drop and re-create the ``db.fs.files`` and ``db.fs.chunks`` collections.
#. Using ``internalClient``, set the following fail point:

   .. code:: javascript

       {
           configureFailPoint: "failCommand",
           mode: { times: 1 },
           data: {
               failCommands: ["insert"],
               blockConnection: true,
               blockTimeMS: 15
           }
       }

#. Create a new MongoClient (referred to as ``client``) with ``timeoutMS=10``.
#. Using ``client``, create a GridFS bucket (referred to as ``bucket``) that wraps the ``db`` database.
#. Call ``bucket.open_upload_stream()`` with the filename ``filename`` to create an upload stream (referred to as ``uploadStream``).

   - Expect this to succeed and return a non-null stream.

#. Using ``uploadStream``, upload a single ``0x12`` byte.
#. Call ``uploadStream.close()`` to flush the stream and insert chunks.

   - Expect this to fail with a timeout error.

Aborting an upload stream can be timed out
``````````````````````````````````````````

This test only applies to drivers that provide an API to abort a GridFS upload stream.

#. Using ``internalClient``, drop and re-create the ``db.fs.files`` and ``db.fs.chunks`` collections.
#. Using ``internalClient``, set the following fail point:

   .. code:: javascript

       {
           configureFailPoint: "failCommand",
           mode: { times: 1 },
           data: {
               failCommands: ["delete"],
               blockConnection: true,
               blockTimeMS: 15
           }
       }

#. Create a new MongoClient (referred to as ``client``) with ``timeoutMS=10``.
#. Using ``client``, create a GridFS bucket (referred to as ``bucket``) that wraps the ``db`` database with ``chunkSizeBytes=2``.
#. Call ``bucket.open_upload_stream()`` with the filename ``filename`` to create an upload stream (referred to as ``uploadStream``).

   - Expect this to succeed and return a non-null stream.

#. Using ``uploadStream``, upload the bytes ``[0x01, 0x02, 0x03, 0x04]``.
#. Call ``uploadStream.abort()``.

   - Expect this to fail with a timeout error.

GridFS - Download
~~~~~~~~~~~~~~~~~

This test MUST only be run against server versions 4.4 and higher.

#. Using ``internalClient``, drop and re-create the ``db.fs.files`` and ``db.fs.chunks`` collections.
#. Using ``internalClient``, insert the following document into the ``db.fs.files`` collection:

   .. code:: javascript

       {
          "_id": {
            "$oid": "000000000000000000000005"
          },
          "length": 10,
          "chunkSize": 4,
          "uploadDate": {
            "$date": "1970-01-01T00:00:00.000Z"
          },
          "md5": "57d83cd477bfb1ccd975ab33d827a92b",
          "filename": "length-10",
          "contentType": "application/octet-stream",
          "aliases": [],
          "metadata": {}
       }

#. Create a new MongoClient (referred to as ``client``) with ``timeoutMS=10``.
#. Using ``client``, create a GridFS bucket (referred to as ``bucket``) that wraps the ``db`` database.
#. Call ``bucket.open_download_stream`` with the id ``{ "$oid": "000000000000000000000005" }`` to create a download stream (referred to as ``downloadStream``).

   - Expect this to succeed and return a non-null stream.

#. Using ``internalClient``, set the following fail point:

   .. code:: javascript

       {
           configureFailPoint: "failCommand",
           mode: { times: 1 },
           data: {
               failCommands: ["find"],
               blockConnection: true,
               blockTimeMS: 15
           }
       }

#. Read from the ``downloadStream``.

   - Expect this to fail with a timeout error.

#. Verify that two ``find`` commands were executed during the read: one against ``db.fs.files`` and another against ``db.fs.chunks``.

Server Selection
~~~~~~~~~~~~~~~~

serverSelectionTimeoutMS honored if timeoutMS is not set
````````````````````````````````````````````````````````

#. Create a MongoClient (referred to as ``client``) with URI ``mongodb://invalid/?serverSelectionTimeoutMS=10``.

#. Using ``client``, execute the command ``{ ping: 1 }`` against the ``admin`` database.

   - Expect this to fail with a server selection timeout error after no more than 15ms.

timeoutMS honored for server selection if it's lower than serverSelectionTimeoutMS
``````````````````````````````````````````````````````````````````````````````````

#. Create a MongoClient (referred to as ``client``) with URI ``mongodb://invalid/?timeoutMS=10&serverSelectionTimeoutMS=20``.

#. Using ``client``, run the command ``{ ping: 1 }`` against the ``admin`` database.

   - Expect this to fail with a server selection timeout error after no more than 15ms.

serverSelectionTimeoutMS honored for server selection if it's lower than timeoutMS
``````````````````````````````````````````````````````````````````````````````````

#. Create a MongoClient (referred to as ``client``) with URI ``mongodb://invalid/?timeoutMS=20&serverSelectionTimeoutMS=10``.

#. Using ``client``, run the command ``{ ping: 1 }`` against the ``admin`` database.

   - Expect this to fail with a server selection timeout error after no more than 15ms.

serverSelectionTimeoutMS honored for server selection if timeoutMS=0
````````````````````````````````````````````````````````````````````

#. Create a MongoClient (referred to as ``client``) with URI ``mongodb://invalid/?timeoutMS=0&serverSelectionTimeoutMS=10``.

#. Using ``client``, run the command ``{ ping: 1 }`` against the ``admin`` database.

   - Expect this to fail with a server selection timeout error after no more than 15ms.

timeoutMS honored for connection handshake commands if it's lower than serverSelectionTimeoutMS
```````````````````````````````````````````````````````````````````````````````````````````````

This test MUST only be run if the server version is 4.4 or higher and the URI has authentication fields (i.e. a
username and password).

#. Using ``internalClient``, set the following fail point:

   .. code:: javascript

       {
           configureFailPoint: failCommand,
           mode: { times: 1 },
           data: {
               failCommands: ["saslContinue"],
               blockConnection: true,
               blockTimeMS: 15
           }
       }

#. Create a new MongoClient (referred to as ``client``) with ``timeoutMS=10`` and ``serverSelectionTimeoutMS=20``.
#. Using ``client``, insert the document ``{ x: 1 }`` into collection ``db.coll``.

   - Expect this to fail with a timeout error after no more than 15ms.

serverSelectionTimeoutMS honored for connection handshake commands if it's lower than timeoutMS
```````````````````````````````````````````````````````````````````````````````````````````````

This test MUST only be run if the server version is 4.4 or higher and the URI has authentication fields (i.e. a
username and password).

#. Using ``internalClient``, set the following fail point:

   .. code:: javascript

       {
           configureFailPoint: failCommand,
           mode: { times: 1 },
           data: {
               failCommands: ["saslContinue"],
               blockConnection: true,
               blockTimeMS: 15
           }
       }

#. Create a new MongoClient (referred to as ``client``) with ``timeoutMS=20`` and ``serverSelectionTimeoutMS=10``.
#. Using ``client``, insert the document ``{ x: 1 }`` into collection ``db.coll``.

   - Expect this to fail with a timeout error after no more than 15ms.

endSession
~~~~~~~~~~

This test MUST only be run against replica sets and sharded clusters with server version 4.4 or higher. It MUST be
run three times: once with the timeout specified via the MongoClient ``timeoutMS`` option, once with the timeout
specified via the ClientSession ``defaultTimeoutMS`` option, and once more with the timeout specified via the
``timeoutMS`` option for the ``endSession`` operation. In all cases, the timeout MUST be set to 10 milliseconds.

#. Using ``internalClient``, drop the ``db.coll`` collection.
#. Using ``internalClient``, set the following fail point:

   .. code:: javascript

       {
           configureFailPoint: failCommand,
           mode: { times: 1 },
           data: {
               failCommands: ["abortTransaction"],
               blockConnection: true,
               blockTimeMS: 15
           }
       }

#. Create a new MongoClient (referred to as ``client``) and an explicit ClientSession derived from that MongoClient (referred to as ``session``).
#. Execute the following code:

   .. code:: typescript

       coll = client.database("db").collection("coll")
       session.start_transaction()
       coll.insert_one({x: 1}, session=session)

#. Using ``session``, execute ``session.end_session``

   - Expect this to fail with a timeout error after no more than 15ms.

Convenient Transactions
~~~~~~~~~~~~~~~~~~~~~~~

Tests in this section MUST only run against replica sets and sharded clusters with server versions 4.4 or higher.

timeoutMS is refreshed for abortTransaction if the callback fails
`````````````````````````````````````````````````````````````````

#. Using ``internalClient``, drop the ``db.coll`` collection.
#. Using ``internalClient``, set the following fail point:

   .. code:: javascript

       {
           configureFailPoint: failCommand,
           mode: { times: 2 },
           data: {
               failCommands: ["insert", "abortTransaction"],
               blockConnection: true,
               blockTimeMS: 15
           }
       }

#. Create a new MongoClient (referred to as ``client``) configured with ``timeoutMS=10`` and an explicit ClientSession derived from that MongoClient (referred to as ``session``).
#. Using ``session``, execute a ``withTransaction`` operation with the following callback:

   .. code:: typescript

       def callback() {
           coll = client.database("db").collection("coll")
           coll.insert_one({ _id: 1 }, session=session)
       }

#. Expect the previous ``withTransaction`` call to fail with a timeout error.
#. Verify that the following events were published during the ``withTransaction`` call:

   #. ``command_started`` and ``command_failed`` events for an ``insert`` command.
   #. ``command_started`` and ``command_failed`` events for an ``abortTransaction`` command.

Unit Tests
==========

The tests enumerated in this section could not be expressed in either spec or prose format. Drivers SHOULD implement
these if it is possible to do so using the driver's existing test infrastructure.

- Operations should ignore ``waitQueueTimeoutMS`` if ``timeoutMS`` is also set.
- If ``timeoutMS`` is set for an operation, the remaining ``timeoutMS`` value should apply to connection checkout after a server has been selected.
- If ``timeoutMS`` is not set for an operation, ``waitQueueTimeoutMS`` should apply to connection checkout after a server has been selected.
- If a new connection is required to execute an operation, ``min(remaining computedServerSelectionTimeout, connectTimeoutMS)`` should apply to socket establishment.
- For drivers that have control over OCSP behavior, ``min(remaining computedServerSelectionTimeout, 5 seconds)`` should apply to HTTP requests against OCSP responders.
- If ``timeoutMS`` is unset, operations fail after two non-consecutive socket timeouts.
- The remaining ``timeoutMS`` value should apply to HTTP requests against KMS servers for CSFLE.
- The remaining ``timeoutMS`` value should apply to commands sent to mongocryptd as part of automatic encryption.
- When doing ``minPoolSize`` maintenance, ``connectTimeoutMS`` is used as the timeout for socket establishment.
