Getting started with Studio

Account Setup

This tutorial will run online on https://studio.specmatic.com. Login there with your Google account and follow along!

PetStore API Specification

Hover over the little pop-out hamburger shown below, and click on the petstore.yaml specification.

screenshot of the file browser pop-out

Then click on the Spec tab at the top to view the spec, as in the following screenshot.

screenshot of the sample petstore.yaml spec

Here’s a copy of the petstore specification if needed for use later in the tutorial.

Provider Side - Contract as a Test

We have a sample implementation of the PetStore API running which you can access through curl or any other tool of your choice.

curl https://my-json-server.typicode.com/specmatic/specmatic-documentation-examples/pets/1

Now lets use Specmatic to run the petstore.yaml API specification as a contract test against the Provider / API to see if it is adhering the OpenAPI Specification.

  1. Select your specification from the file browser and open the Test tab at the top of the screen.
  2. Just run the tests.

The test will pass.

Notes:

  • The base URL for the backend is pre-filled in the box next to the Run button. It comes from the first url in the spec.
  • You should see 100% API coverage percentage of 100%. That’s because all APIs have been covered (in this case, there’s just one API).
  • Click on Covered in the test results in order to drill down into the tests run for that API.
  • Then click on Success to see more details about the request.

Where did Specmatic get the test data to generate the HTTP request

How did Specmatic know to make the exact request to GET /specmatic/specmatic-documentation-examples/pets/1 with petId as “1”? And not just any other number?

In the OpenAPI spec you may have noticed that there is an examples section for petid with a named example called SCOOBY_200_OK.

  - name: "petid"
    in: "path"
    required: true
    schema:
      type: "number"
    examples:
      SCOOBY_200_OK:
        value: 1

Click on the Spec tab, remove the examples section such that the petid param look as shown below, and save the file.

  - name: "petid"
    in: "path"
    required: true
    schema:
      type: "number"

And try running the tests again.

This results in a test failure because the sample application returns a 404. Drill-down into the test details, and take a closer look at the URL of the test request. Where before we saw GET /specmatic/specmatic-documentation-examples/pets/1 (the value of the petId parameter here is 1), you’ll now see a random petId. Since we removed the examples named SCOOBY_200_OK, Specmatic generated a random example of petId (which is a number in the spec). The request went out this random petId in the path. The application returned a 404 because it has no data for this petId.

Click on Spec tab, restore petstore.yaml back to it’s original state by pasting this content back in, save it, and run the tests. With the example back, the tests will pass once again.

How does this all work?

  • Specmatic is able to tie the named example SCOOBY_200_OK listed under the request parameters and the response sections of the OpenAPI spec to create a test.
  • This is also reflected in the name of the test where Specmatic displays the SCOOBY_200_OK in the test logs
  • Here’s a detailed breakdown of the contract test:
    • Request: Specmatic uses the value defined for the petId request parameter from the SCOOBY_200_OK request example to make a HTTP request.
    • Response: In order to tie the above request with a HTTP response code in the spec, Specmatic looks for an example with same name: SCOOBY_200_OK under responses. In this case the response code happens to be 200. This request/response pair now forms a test case.
    • Response Validation: Note that we are running the specification as a contract test here, in which we are interested in validating only the API signature and not the API logic. Hence, Specmatic does not validate the actual response values defined in the SCOOBY_200_OK example against the values returned by the application. It only validates the response code. However, if you do wish to validate response values, you can find more details in our discussion here.

What happens when OpenAPI goes out of sync with the application or vice versa?

Restore the original content of the spec back, using the Spec tab. You can copy the original spec from here, paste it into the Spec tab, and save it.

Now lets try something more interesting. Change the datatype of the status field of response in OpenAPI file to boolean and save it.

  properties:
    status:
      type: "boolean"

Also modify the status field in the SCOOBY_200_OK example just below it, like so:

  examples:
    SCOOBY_200_OK:
      value:
        id: 1
        name: Scooby
        type: Golden Retriever
        status: true # change the value from "Adopted" to true

Now try to run the tests. You’ll see a red error message explaining how status is an enum, and true is not a valid value.

This is how Specmatic is able to make sure that your API never deviates from the Specification.

Please refer to below videos for extensive demos on Contract Tests.

Learn more about Contract Tests here.

Consumer Side - Contract As A Mock / Intelligent Service Virtualisation

To spin up a mock server with petstore.yaml, go to the Mock tab and press the Run button.

This should start your mock server on a random port.

Once the mock server is running you can verify the API by accessing it through Postman, Chrome, Curl etc.

curl https://studio.specmatic.io:<port number>/pets/123

You should now be able to see the response that matches the schema defined in your OpenAPI spec.

{
    "id": 864,
    "name": "VRIQA",
    "type": "KPNDQ",
    "status": "QQSFF"
}

The response contains auto-generated values that adhere to the data type defined in the contract. In above output petid “864” is generated by specmatic and will vary with every execution.

Also try making a request with petId 1:

curl https://studio.specmatic.io:<port number>/pets/1

It will always return below values:

{
    "id": 1,
    "name": "Scooby",
    "type": "Golden Retriever",
    "status": "Adopted"
}

This is thanks to the example SCOOBY_200_OK in the petstore.yaml spec file, which we earlier saw being used while running contract test. Specmatic also uses it to serves a mock response.

With this we have effectively achived three goals in one go.

  • Examples serve as sample data for people referring to the API specification as documentation
  • The same examples are used in contract tests to create the HTTP request
  • And these examples also serve as mock data when we run Specmatic mock command

Intelligent Service Virtualisation

Let us try a few experiments. Remove the status field in the 200_OKAY response example in petstore.yaml (the very last line in that file) and run the mock command again.

examples:
  200_OKAY:
    value:
      id: 1
      type: "Golden Retriever"
      name: "Scooby"
      status: "Adopted" # Remove this line

The mock server will auto reload your petstore.yaml file as soon as you save it. And you should see a warning in a yellow box on the Mock tab saying this:

[Example SCOOBY_200_OK]: Error from contract `petstore.yaml`

  In scenario "Should be able to get a pet by petId. Response: Should be able to get a pet by petId"
  API: GET /pets/(petid:number) -> 200

    >> RESPONSE.BODY.status

       key named status in the spec was not found in the "SCOOBY_200_OK" example

Specmatic rejects the expectation / canned response since it is not in line with the OpenAPI Specification.

Externalising mock responses

Please restore petstore.yaml to its original state before proceeding with this section.

If you would like to add more mock responses, however you do not wish to bloat your specification with a lot of examples, we can also externalise the mock / canned responses to json files also.

Go to the Examples tab. Press generate. The file here runs exists outside the spec. Click on the filename, and update the values in the example as follows:

  • Set path tp “/pets/2”
  • Add the following in the response body: 
      "id": 2,
  "name": "Togo",
  "type": "Siberian Husky",
  "status": "Adopted"

Now run the mock again.

Once the mock server is running you can verify the API by accessing it through Postman, Chrome, Curl etc.

curl http://localhost:9000/pets/2

You should now be able to see the data pertaining to the togo.json file that you added.

{
    "id": 2,
    "name": "Togo",
    "type": "Siberian Husky",
    "status": "Adopted"
}

Specmatic validates this externalised mock JSON file togo.json against the petstore.yaml. Let us try this by removing the status field within http-response body in togo.json and run the mock command again.

You should see a warning as follows.

 Example files in './../../../specs/service_examples'
  - /specs/service_examples/example.json

  >> Error loading mock expectation file '/specs/service_examples/example.json':
   /specs/service_examples/example.json didn't match /specs/petstore.yaml
    Error from contract /specs/petstore.yaml

      In scenario "Should be able to get a pet by petId. Response: Should be able to get a pet by petId"
      API: GET /pets/(petid:number) -> 200

        >> RESPONSE.BODY.status

           Key named status in the contract was not found in the mock

Specmatic again rejects the expectation / canned response since it is not in line with the OpenAPI Specification.

We can now start consumer development against this mock without any dependency on the real API.

To know more about Intelligent Service Virtualisation please refer to below video demos

Learn more about Mocking / Smart Mocks here.

A more complicated example

A sample application is available in Studio at https://order-bff:8080 to try out. It’s accessible only from Studio. You can run contract tests against it using the specification named product_search_bff_v3.yaml also shipped to your account by default.

Try running Generative tests to see what happens.