Test Configuration

Resiliency Tests

To enable resiliency tests, add the resiliencyTests configuration to your specmatic.yaml file as shown below:

Version 3 (recommended)

YAML

specmatic.yaml

version: 3
systemUnderTest:
  service:
    $ref: "#/components/services/petStore"
    runOptions:
      $ref: "#/components/runOptions/petStore"
components:
  services:
    petStore:
      description: Pet Store Service
      definitions:
        - definition:
            source:
              filesystem:
                directory: path/to/spec/directory
            specs:
              - com/petstore/store.yaml
  settings:
    test:
      schemaResiliencyTests: all
  runOptions:
    petStore:
      openapi:
        type: test

specmatic.json

specmatic.json

{
  "version": 3,
  "systemUnderTest": {
    "service": {
      "$ref": "#/components/services/petStore",
      "runOptions": {
        "$ref": "#/components/runOptions/petStore"
      }
    }
  },
  "components": {
    "services": {
      "petStore": {
        "description": "Pet Store Service",
        "definitions": [
          {
            "definition": {
              "source": {
                "filesystem": {
                  "directory": "path/to/spec/directory"
                }
              },
              "specs": [
                "com/petstore/store.yaml"
              ]
            }
          }
        ]
      }
    },
    "settings": {
      "test": {
        "schemaResiliencyTests": "all"
      }
    },
    "runOptions": {
      "petStore": {
        "openapi": {
          "type": "test"
        }
      }
    }
  }
}

Version 2

YAML

specmatic.yaml

version: 2
contracts:
  - provides:
      - /path/to/your/spec.yaml
test:
  resiliencyTests:
    enable: all

JSON

specmatic.json

{
  "version": 2,
  "contracts": [
    {
      "provides": [
        "/path/to/your/spec.yaml"
      ]
    }
  ],
  "test": {
    "resiliencyTests": {
      "enable": "all"
    }
  }
}

note

The resiliency test can be configured with the following options:

• all — will run both positive and negative tests
• positiveOnly — will not run negative tests
• none - the default

info

When using configuration version 3, it is possible to set the schemaResiliencyTests option under dependencies, or systemUnderTest or within a specific service. This allows you to enable resiliency tests for specific dependencies or the system under test, while keeping it disabled for other services.

Contract Test Timeout

The HTTP timeout duration for requests made during contract testing can be configured using timeoutInMilliseconds parameter. This parameter sets the maximum time Specmatic will wait for a response to each HTTP request before marking it as a failure. The default timeout is 6000 milliseconds.

Version 3 (recommended)

YAML

specmatic.yaml

version: 3
dependencies:
  services:
    - service:
        $ref: "#/components/services/apiOrderV3"
components:
  sources:
    specmaticOrderContracts:
      git:
        url: https://github.com/specmatic/specmatic-order-contracts.git
  services:
    apiOrderV3:
      description: API Order V3
      definitions:
        - definition:
            source:
              $ref: "#/components/sources/specmaticOrderContracts"
            specs:
              - io/specmatic/examples/store/openapi/api_order_v3.yaml
specmatic:
  settings:
    test:
      timeoutInMilliseconds: 3000

specmatic.json

specmatic.json

{
  "version": 3,
  "dependencies": {
    "services": [
      {
        "service": {
          "$ref": "#/components/services/apiOrderV3"
        }
      }
    ]
  },
  "components": {
    "sources": {
      "specmaticOrderContracts": {
        "git": {
          "url": "https://github.com/specmatic/specmatic-order-contracts.git"
        }
      }
    },
    "services": {
      "apiOrderV3": {
        "description": "API Order V3",
        "definitions": [
          {
            "definition": {
              "source": {
                "$ref": "#/components/sources/specmaticOrderContracts"
              },
              "specs": [
                "io/specmatic/examples/store/openapi/api_order_v3.yaml"
              ]
            }
          }
        ]
      }
    }
  },
  "specmatic": {
    "settings": {
      "test": {
        "timeoutInMilliseconds": 3000
      }
    }
  }
}

Version 2

YAML

specmatic.yaml

version: 2
contracts:
  - git:
      url: https://github.com/specmatic/specmatic-order-contracts.git
    consumes:
      - io/specmatic/examples/store/openapi/api_order_v3.yaml
test:
  timeoutInMilliseconds: 3000

JSON

specmatic.json

{
  "version": 2,
  "contracts": [
    {
      "git": {
        "url": "https://github.com/specmatic/specmatic-order-contracts.git"
      },
      "consumes": [
        "io/specmatic/examples/store/openapi/api_order_v3.yaml"
      ]
    }
  ],
  "test": {
    "timeoutInMilliseconds": 3000
  }
}

AsyncAPI Test Configuration

AsyncAPI contract tests use runOptions.asyncapi in specmatic.yaml. This is where Specmatic reads the async test settings represented by AsyncTestConfig.

YAML

specmatic.yaml

version: 3
systemUnderTest:
  service:
    definitions:
      - definition:
          source:
            filesystem:
              directory: api-specs
          specs:
            - order-service-async-avro-v3_0_0.yaml
    runOptions:
      asyncapi:
        type: test
        replyTimeout: 15000
        subscriberReadinessWaitTime: 3000
        schemaRegistry:
          kind: CONFLUENT
          url: ${SCHEMA_REGISTRY_URL:http://localhost:8085}
          username: admin
          password: admin-secret
        servers:
          - host: ${KAFKA_BROKER:localhost:9092}
            protocol: kafka
            adminCredentials:
              security.protocol: SASL_PLAINTEXT
              sasl.mechanism: PLAIN
              sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="test" password="test-secret";
            client:
              producer:
                basic.auth.credentials.source: USER_INFO
                basic.auth.user.info: admin:admin-secret
              consumer:
                basic.auth.credentials.source: USER_INFO
                basic.auth.user.info: admin:admin-secret

JSON

specmatic.json

{
  "version": 3,
  "systemUnderTest": {
    "service": {
      "definitions": [
        {
          "definition": {
            "source": {
              "filesystem": {
                "directory": "api-specs"
              }
            },
            "specs": [
              "order-service-async-avro-v3_0_0.yaml"
            ]
          }
        }
      ],
      "runOptions": {
        "asyncapi": {
          "type": "test",
          "replyTimeout": 15000,
          "subscriberReadinessWaitTime": 3000,
          "schemaRegistry": {
            "kind": "CONFLUENT",
            "url": "${SCHEMA_REGISTRY_URL:http://localhost:8085}",
            "username": "admin",
            "password": "admin-secret"
          },
          "servers": [
            {
              "host": "${KAFKA_BROKER:localhost:9092}",
              "protocol": "kafka",
              "adminCredentials": {
                "security.protocol": "SASL_PLAINTEXT",
                "sasl.mechanism": "PLAIN",
                "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"test\" password=\"test-secret\";"
              },
              "client": {
                "producer": {
                  "basic.auth.credentials.source": "USER_INFO",
                  "basic.auth.user.info": "admin:admin-secret"
                },
                "consumer": {
                  "basic.auth.credentials.source": "USER_INFO",
                  "basic.auth.user.info": "admin:admin-secret"
                }
              }
            }
          ]
        }
      }
    }
  }
}

AsyncAPI Test Options

Property	Description
type	Must be test for AsyncAPI contract tests.
replyTimeout	Maximum time in milliseconds to wait for a reply message. Default: 10000.
subscriberReadinessWaitTime	Additional wait time in milliseconds before tests begin verifying messages. Use this when consumers or subscriptions need time to become ready. Default: 0.
servers	List of broker endpoints Specmatic can use while running the test. Specmatic matches entries by protocol.
servers[].host	Broker host or endpoint. For Kafka, this is typically host:port.
servers[].protocol	Messaging protocol such as kafka, sqs, sns, jms, or rabbitmq.
servers[].adminCredentials	Protocol-specific admin properties used to provision or inspect broker resources.
servers[].client.producer	Producer client properties passed through to the protocol client.
servers[].client.consumer	Consumer client properties passed through to the protocol client.
schemaRegistry	Optional schema registry settings for Avro-based AsyncAPI contracts.
schemaRegistry.kind	Schema registry type. Currently CONFLUENT is supported.
schemaRegistry.url	Schema registry endpoint URL.
schemaRegistry.username	Optional username for the schema registry.
schemaRegistry.password	Optional password for the schema registry.

Notes

• Add replyTimeout and subscriberReadinessWaitTime only when the defaults are not enough for your environment.
• adminCredentials, client.producer, and client.consumer are protocol-specific maps. For Kafka, use standard Kafka client properties such as security.protocol, sasl.mechanism, and serializer or auth settings.
• schemaRegistry is only needed when your AsyncAPI payloads reference schemas from a registry, such as the Kafka Avro sample.
• For Kafka-specific examples, see Kafka.

Strict Mode

Test may run in strict mode. You can read more about test in strict mode here.

You can configure Specmatic to run test in strict mode using the following configuration:

Version 3 (recommended)

YAML

specmatic.yaml

version: 3
dependencies:
  services:
    - service:
        $ref: "#/components/services/apiOrderV3"
components:
  sources:
    specmaticOrderContracts:
      git:
        url: https://github.com/specmatic/specmatic-order-contracts.git
  services:
    apiOrderV3:
      description: API Order V3
      definitions:
        - definition:
            source:
              $ref: "#/components/sources/specmaticOrderContracts"
            specs:
              - io/specmatic/examples/store/openapi/api_order_v3.yaml
specmatic:
  settings:
    test:
      strictMode: true

specmatic.json

specmatic.json

{
  "version": 3,
  "dependencies": {
    "services": [
      {
        "service": {
          "$ref": "#/components/services/apiOrderV3"
        }
      }
    ]
  },
  "components": {
    "sources": {
      "specmaticOrderContracts": {
        "git": {
          "url": "https://github.com/specmatic/specmatic-order-contracts.git"
        }
      }
    },
    "services": {
      "apiOrderV3": {
        "description": "API Order V3",
        "definitions": [
          {
            "definition": {
              "source": {
                "$ref": "#/components/sources/specmaticOrderContracts"
              },
              "specs": [
                "io/specmatic/examples/store/openapi/api_order_v3.yaml"
              ]
            }
          }
        ]
      }
    }
  },
  "specmatic": {
    "settings": {
      "test": {
        "strictMode": true
      }
    }
  }
}

Version 2

YAML

specmatic.yaml

version: 2
contracts:
  - provides:
      - /path/to/spec.yaml
test:
  strictMode: true

JSON

specmatic.json

{
  "version": 2,
  "contracts": [
    {
      "provides": [
        "/path/to/spec.yaml"
      ]
    }
  ],
  "test": {
    "strictMode": true
  }
}

Configure Contract Test Base URLs for Different Specs

Let's say you have a service that two specifications, or even multiple versions of the same specification. You need to run contract tests using that specification against the service. You can customize the base URL in specmatic configuration as shown below.

Version 3 (recommended)

YAML

specmatic.yaml

version: 3
dependencies:
  services:
    - service:
        $ref: "#/components/services/service-v3"
        runOptions:
          $ref: "#/components/runOptions/service-v3"
    - service:
        $ref: "#/components/services/service-v4"
        runOptions:
          $ref: "#/components/runOptions/service-v4"
components:
  sources:
    specmaticOrderContracts:
      git:
        url: https://github.com/specmatic/specmatic-order-contracts.git
  services:
    service-v3:
      description: V3 of the service
      definitions:
        - definition:
            source:
              $ref: "#/components/sources/specmaticOrderContracts"
            specs:
              - io/specmatic/examples/store/openapi/api_order_v3.yaml
    service-v4:
      description: V4 of the service
      definitions:
        - definition:
            source:
              $ref: "#/components/sources/specmaticOrderContracts"
            specs:
              - io/specmatic/examples/store/openapi/api_order_v4.yaml
  runOptions:
    service-v3:
      openapi:
        type: mock
        baseUrl: http://localhost:8080/v3
    service-v4:
      openapi:
        type: mock
        baseUrl: http://localhost:8080/v4

specmatic.json

specmatic.json

{
  "version": 3,
  "dependencies": {
    "services": [
      {
        "service": {
          "$ref": "#/components/services/service-v3",
          "runOptions": {
            "$ref": "#/components/runOptions/service-v3"
          }
        }
      },
      {
        "service": {
          "$ref": "#/components/services/service-v4",
          "runOptions": {
            "$ref": "#/components/runOptions/service-v4"
          }
        }
      }
    ]
  },
  "components": {
    "sources": {
      "specmaticOrderContracts": {
        "git": {
          "url": "https://github.com/specmatic/specmatic-order-contracts.git"
        }
      }
    },
    "services": {
      "service-v3": {
        "description": "V3 of the service",
        "definitions": [
          {
            "definition": {
              "source": {
                "$ref": "#/components/sources/specmaticOrderContracts"
              },
              "specs": [
                "io/specmatic/examples/store/openapi/api_order_v3.yaml"
              ]
            }
          }
        ]
      },
      "service-v4": {
        "description": "V4 of the service",
        "definitions": [
          {
            "definition": {
              "source": {
                "$ref": "#/components/sources/specmaticOrderContracts"
              },
              "specs": [
                "io/specmatic/examples/store/openapi/api_order_v4.yaml"
              ]
            }
          }
        ]
      }
    },
    "runOptions": {
      "service-v3": {
        "openapi": {
          "type": "mock",
          "baseUrl": "http://localhost:8080/v3"
        }
      },
      "service-v4": {
        "openapi": {
          "type": "mock",
          "baseUrl": "http://localhost:8080/v4"
        }
      }
    }
  }
}

Version 2

YAML

specmatic.yaml

version: 2
contracts:
  - git:
      url: https://github.com/specmatic/specmatic-order-contracts.git
    consumes:
      - specs:
          - io/specmatic/examples/store/openapi/api_order_v3.yaml
        baseUrl: http://localhost:8080/v3
      - specs:
          - io/specmatic/examples/store/openapi/api_order_v4.yaml
        baseUrl: http://localhost:8080/v4

JSON

specmatic.json

{
  "version": 2,
  "contracts": [
    {
      "git": {
        "url": "https://github.com/specmatic/specmatic-order-contracts.git"
      },
      "consumes": [
        {
          "specs": [
            "io/specmatic/examples/store/openapi/api_order_v3.yaml"
          ],
          "baseUrl": "http://localhost:8080/v3"
        },
        {
          "specs": [
            "io/specmatic/examples/store/openapi/api_order_v4.yaml"
          ],
          "baseUrl": "http://localhost:8080/v4"
        }
      ]
    }
  ]
}

Workflow

You can configure workflow rules to extract data from one API response and use it in subsequent requests during contract testing.

Version 3 (recommended)

YAML

specmatic.yaml

version: 3
systemUnderTest:
  service:
    $ref: "#/components/services/orderApiService"
    runOptions:
      $ref: "#/components/runOptions/orderApiTest"
components:
  sources:
    localSpecs:
      filesystem:
        directory: .
  services:
    orderApiService:
      description: Order API Service
      definitions:
        - definition:
            source:
              $ref: "#/components/sources/localSpecs"
            specs:
              - ./openapi/order.yaml
  runOptions:
    orderApiTest:
      openapi:
        type: test
        baseUrl: http://localhost:8080
        workflow:
          ids:
            POST /orders -> 201:
              extract: BODY.id
            "*":
              use: PATH.orderId

specmatic.json

specmatic.json

{
  "version": 3,
  "systemUnderTest": {
    "service": {
      "$ref": "#/components/services/orderApiService",
      "runOptions": {
        "$ref": "#/components/runOptions/orderApiTest"
      }
    }
  },
  "components": {
    "sources": {
      "localSpecs": {
        "filesystem": {
          "directory": "."
        }
      }
    },
    "services": {
      "orderApiService": {
        "description": "Order API Service",
        "definitions": [
          {
            "definition": {
              "source": {
                "$ref": "#/components/sources/localSpecs"
              },
              "specs": [
                "./openapi/order.yaml"
              ]
            }
          }
        ]
      }
    },
    "runOptions": {
      "orderApiTest": {
        "openapi": {
          "type": "test",
          "baseUrl": "http://localhost:8080",
          "workflow": {
            "ids": {
              "POST /orders -> 201": {
                "extract": "BODY.id"
              },
              "*": {
                "use": "PATH.orderId"
              }
            }
          }
        }
      }
    }
  }
}

Version 2

YAML

specmatic.yaml

version: 2
contracts:
  - provides:
      - ./openapi/order.yaml
workflow:
  ids:
    POST /orders -> 201:
      extract: BODY.id
    "*":
      use: PATH.orderId

JSON

specmatic.json

{
  "version": 2,
  "contracts": [
    {
      "provides": [
        "./openapi/order.yaml"
      ]
    }
  ],
  "workflow": {
    "ids": {
      "POST /orders -> 201": {
        "extract": "BODY.id"
      },
      "*": {
        "use": "PATH.orderId"
      }
    }
  }
}

Externalized Examples Directories

By default, Specmatic searches for the directory ending with _examples to pickup externalized examples. However, if needed, you can specify a list of directories containing externalized examples under examples key in specmatic configuration. Specmatic will retrieve the examples from these directories for use in both contract testing and service virtualization.

Version 3 (recommended)

YAML

specmatic.yaml

version: 3
systemUnderTest:
  service:
    $ref: "#/components/services/productSearchBffV4"
    runOptions:
      openapi:
        type: mock
        host: localhost
        port: 9000
dependencies:
  services:
    - service:
        $ref: "#/components/services/apiOrderV3"
        runOptions:
          openapi:
            type: mock
            host: localhost
            port: 9001
components:
  sources:
    specmaticOrderContracts:
      git:
        url: https://github.com/specmatic/specmatic-order-contracts.git
  services:
    productSearchBffV4:
      description: Product Search BFF v4 (provider)
      definitions:
        - definition:
            source:
              $ref: "#/components/sources/specmaticOrderContracts"
            specs:
              - io/specmatic/examples/store/openapi/product_search_bff_v4.yaml
    apiOrderV3:
      description: API Order V3 (consumer)
      definitions:
        - definition:
            source:
              $ref: "#/components/sources/specmaticOrderContracts"
            specs:
              - io/specmatic/examples/store/openapi/api_order_v3.yaml
  examples:
    commonExamples:
      directories:
        - order_service/examples
        - product_service/examples

specmatic.json

specmatic.json

{
  "version": 3,
  "systemUnderTest": {
    "service": {
      "$ref": "#/components/services/productSearchBffV4",
      "runOptions": {
        "openapi": {
          "type": "mock",
          "host": "localhost",
          "port": 9000
        }
      }
    }
  },
  "dependencies": {
    "services": [
      {
        "service": {
          "$ref": "#/components/services/apiOrderV3",
          "runOptions": {
            "openapi": {
              "type": "mock",
              "host": "localhost",
              "port": 9001
            }
          }
        }
      }
    ]
  },
  "components": {
    "sources": {
      "specmaticOrderContracts": {
        "git": {
          "url": "https://github.com/specmatic/specmatic-order-contracts.git"
        }
      }
    },
    "services": {
      "productSearchBffV4": {
        "description": "Product Search BFF v4 (provider)",
        "definitions": [
          {
            "definition": {
              "source": {
                "$ref": "#/components/sources/specmaticOrderContracts"
              },
              "specs": [
                "io/specmatic/examples/store/openapi/product_search_bff_v4.yaml"
              ]
            }
          }
        ]
      },
      "apiOrderV3": {
        "description": "API Order V3 (consumer)",
        "definitions": [
          {
            "definition": {
              "source": {
                "$ref": "#/components/sources/specmaticOrderContracts"
              },
              "specs": [
                "io/specmatic/examples/store/openapi/api_order_v3.yaml"
              ]
            }
          }
        ]
      }
    },
    "examples": {
      "commonExamples": {
        "directories": [
          "order_service/examples",
          "product_service/examples"
        ]
      }
    }
  }
}

Version 2

YAML

specmatic.yaml

version: 2
contracts:
  - git:
      url: https://github.com/specmatic/specmatic-order-contracts.git
    provides:
      - io/specmatic/examples/store/openapi/product_search_bff_v4.yaml
    consumes:
      - io/specmatic/examples/store/openapi/api_order_v3.yaml
examples:
  - order_service/examples
  - product_service/examples

JSON

specmatic.json

{
  "version": 2,
  "contracts": [
    {
      "git": {
        "url": "https://github.com/specmatic/specmatic-order-contracts.git"
      },
      "provides": [
        "io/specmatic/examples/store/openapi/product_search_bff_v4.yaml"
      ],
      "consumes": [
        "io/specmatic/examples/store/openapi/api_order_v3.yaml"
      ]
    }
  ],
  "examples": [
    "order_service/examples",
    "product_service/examples"
  ]
}

note

When using v2 config, if the _examples directory is present, it will still be included alongside any additional directories specified under the examples key.

When using v3 config, the examples will be loaded from directory with the name ${SPEC_FILE}_examples if it exists, in addition to any directories specified under the examples key. For example, if you have a spec file named api_order_v1.yaml, Specmatic will look for examples in the api_order_v1_examples directory by default, even if it's not explicitly mentioned in the configuration.