Skip to main content

Test Configuration

Resiliency Tests​

To enable resiliency tests, add the resiliencyTests configuration to your specmatic.yaml file as shown below:

specmatic.yaml
version: 3
systemUnderTest:
  service:
    $ref: "#/components/services/petStore"
    runOptions:
      $ref: "#/components/runOptions/petStore"
components:
  services:
    petStore:
      description: Pet Store Service
      definitions:
        - definition:
            source:
              filesystem:
                directory: path/to/spec/directory
            specs:
              - com/petstore/store.yaml
  settings:
    test:
      schemaResiliencyTests: all
  runOptions:
    petStore:
      openapi:
        type: test
note

The resiliency test can be configured with the following options:

  • all — will run both positive and negative tests
  • positiveOnly — will not run negative tests
  • none - the default
info

When using configuration version 3, it is possible to set the schemaResiliencyTests option under dependencies, or systemUnderTest or within a specific service. This allows you to enable resiliency tests for specific dependencies or the system under test, while keeping it disabled for other services.

Contract Test Timeout​

The HTTP timeout duration for requests made during contract testing can be configured using timeoutInMilliseconds parameter. This parameter sets the maximum time Specmatic will wait for a response to each HTTP request before marking it as a failure. The default timeout is 6000 milliseconds.

specmatic.yaml
version: 3
dependencies:
  services:
    - service:
        $ref: "#/components/services/apiOrderV3"
components:
  sources:
    specmaticOrderContracts:
      git:
        url: https://github.com/specmatic/specmatic-order-contracts.git
  services:
    apiOrderV3:
      description: API Order V3
      definitions:
        - definition:
            source:
              $ref: "#/components/sources/specmaticOrderContracts"
            specs:
              - io/specmatic/examples/store/openapi/api_order_v3.yaml
specmatic:
  settings:
    test:
      timeoutInMilliseconds: 3000

Strict Mode​

Test may run in strict mode. You can read more about test in strict mode here.

You can configure Specmatic to run test in strict mode using the following configuration:

specmatic.yaml
version: 3
dependencies:
  services:
    - service:
        $ref: "#/components/services/apiOrderV3"
components:
  sources:
    specmaticOrderContracts:
      git:
        url: https://github.com/specmatic/specmatic-order-contracts.git
  services:
    apiOrderV3:
      description: API Order V3
      definitions:
        - definition:
            source:
              $ref: "#/components/sources/specmaticOrderContracts"
            specs:
              - io/specmatic/examples/store/openapi/api_order_v3.yaml
specmatic:
  settings:
    test:
      strictMode: true

Configure Contract Test Base URLs for Different Specs​

Let's say you have a service that two specifications, or even multiple versions of the same specification. You need to run contract tests using that specification against the service. You can customize the base URL in specmatic configuration as shown below.

specmatic.yaml
version: 3
dependencies:
  services:
    - service:
        $ref: "#/components/services/service-v3"
        runOptions:
          $ref: "#/components/runOptions/service-v3"
    - service:
        $ref: "#/components/services/service-v4"
        runOptions:
          $ref: "#/components/runOptions/service-v4"
components:
  sources:
    specmaticOrderContracts:
      git:
        url: https://github.com/specmatic/specmatic-order-contracts.git
  services:
    service-v3:
      description: V3 of the service
      definitions:
        - definition:
            source:
              $ref: "#/components/sources/specmaticOrderContracts"
            specs:
              - io/specmatic/examples/store/openapi/api_order_v3.yaml
    service-v4:
      description: V4 of the service
      definitions:
        - definition:
            source:
              $ref: "#/components/sources/specmaticOrderContracts"
            specs:
              - io/specmatic/examples/store/openapi/api_order_v4.yaml
  runOptions:
    service-v3:
      openapi:
        type: mock
        baseUrl: http://localhost:8080/v3
    service-v4:
      openapi:
        type: mock
        baseUrl: http://localhost:8080/v4

Workflow​

You can configure workflow rules to extract data from one API response and use it in subsequent requests during contract testing.

specmatic.yaml
version: 3
systemUnderTest:
  service:
    $ref: "#/components/services/orderApiService"
    runOptions:
      $ref: "#/components/runOptions/orderApiTest"
components:
  sources:
    localSpecs:
      filesystem:
        directory: .
  services:
    orderApiService:
      description: Order API Service
      definitions:
        - definition:
            source:
              $ref: "#/components/sources/localSpecs"
            specs:
              - ./openapi/order.yaml
  runOptions:
    orderApiTest:
      openapi:
        type: test
        baseUrl: http://localhost:8080
        workflow:
          ids:
            POST /orders -> 201:
              extract: BODY.id
            "*":
              use: PATH.orderId

Externalized Examples Directories​

By default, Specmatic searches for the directory ending with _examples to pickup externalized examples. However, if needed, you can specify a list of directories containing externalized examples under examples key in specmatic configuration. Specmatic will retrieve the examples from these directories for use in both contract testing and service virtualization.

specmatic.yaml
version: 3
systemUnderTest:
  service:
    $ref: "#/components/services/productSearchBffV4"
    runOptions:
      openapi:
        type: mock
        host: localhost
        port: 9000
dependencies:
  services:
    - service:
        $ref: "#/components/services/apiOrderV3"
        runOptions:
          openapi:
            type: mock
            host: localhost
            port: 9001
components:
  sources:
    specmaticOrderContracts:
      git:
        url: https://github.com/specmatic/specmatic-order-contracts.git
  services:
    productSearchBffV4:
      description: Product Search BFF v4 (provider)
      definitions:
        - definition:
            source:
              $ref: "#/components/sources/specmaticOrderContracts"
            specs:
              - io/specmatic/examples/store/openapi/product_search_bff_v4.yaml
    apiOrderV3:
      description: API Order V3 (consumer)
      definitions:
        - definition:
            source:
              $ref: "#/components/sources/specmaticOrderContracts"
            specs:
              - io/specmatic/examples/store/openapi/api_order_v3.yaml
  examples:
    commonExamples:
      directories:
        - order_service/examples
        - product_service/examples
note

When using v2 config, if the _examples directory is present, it will still be included alongside any additional directories specified under the examples key.

When using v3 config, the examples will be loaded from directory with the name ${SPEC_FILE}_examples if it exists, in addition to any directories specified under the examples key. For example, if you have a spec file named api_order_v1.yaml, Specmatic will look for examples in the api_order_v1_examples directory by default, even if it's not explicitly mentioned in the configuration.