API Testing 101

I have covered the BINMEN heuristic in more detail here

Whether you developed the API or have been asked to test it, fundamentally you should want to know as much information about it as possible.

The definition of testing I use comes from Michael Bolton. “Testing is an evaluation of a product through experimentation, experience and exploration”, so by exploring, I am referring to testing.

Testing is exploratory, because testers have agency and can change direction, tactics, look at different things based on their intuition, skills, heuristics. When testing, we should be looking for problems and not focusing on successful outcomes.

Exploring an API could also be described as "simultaneous test design, test execution and learning". You are building a model in real-time by experimenting with the API and then taking your learnings as information to guide you through. You may have documentation or requirements that you'e been given. They are claims made about the sofwtare and only through testing those claims can you compare their claims with reality. You may have no requirements to test against and in this case you build your model based on exploration and then report what happened during your experiments.

I’d recommend using session based test management (SBTM) to document your testing. A session is a focused time box with a mission to find some information about the system you want to test.

This is what a charter looks like.

Charter:
  Mission: Try and see user data using invalid API tokens
  Tester: David Jones
Notes:
  API uses basic auth (why not something more secure?)
  Error message when not passing in key is confusing
  When using an expired token, I got a different error code
  
  
Issues:
Wrong status code on expired token
Bugs:

• What is the purpose of the API and how does it achieve that purpose or solve a problem?
• What documentation is available?
• What boundaries does the developer think exist? Does the PO agree?
• What are the requirements and who do they affect?
• What authentication does the API use?
• What happens if I don't pass in a valid token? expired token? invalid token?
• What harm can be caused by using this API improperly? bad auth? data leakage? slow performance?

A test oracle is a "means by which you attempt to recognise a bug during testing" (from "Taking testing seriously" by Bach, Bolton)

Here are some examples of oracles you can look for.

History

• Has the behaviour changed since its last release?
• Is there an older version that we can compare to?
• Is there a way to understand a regression?

Claims

• Does the API do what the developer thinks it does and under what conditions?
• Do you have access to the developers that wrote the code?
• Can you speak directly to the product owner or clients of the product?
• Are there any documents that could help such as guides, swagger, existing customer docs etc?
• What must always happen to this API? (if anything)
• What must never happen to this API? (if anything)

Standards and Regulations

• Does the API need to meet any regulatory standards?
• Does the API meet REST standards?
• Are there implicit or explicit performance obligations? How long is acceptable for a response?
• Is there a requirement for up-time or outage resolution?
• Is there a health check endpoint?
• Is it secure over https?

Comparable / market

• What is a similar product that you can compare to?
• What do competitors offer that this product doesn't?

Performance

• Are cache control or Etags used?
• What kind of traffic does the API receive? Is the traffic in short peaks?
• What does scaling look like?

Test coverage can mean many things. You may have heard of metrics such as branch coverage, line coverage, etc. Even if these types of coverage claim 100% of something, there could still be issues in the code. When testing an API you should be concerned with what you haven't yet tested as well as what you have tested. Therefore, I think it's better to use a heuristic framework rather than deadset coverage rules to think about while you're testing.

To understand an API you should create a product coverage outline of all testable elements of an API

You can use a mindmap or even a list, but the idea is to not take too much time developing this and to do it yourself. This outline feeds into your model and understanding of the product that will inform further, deeper exploration into its features if required

more in-depth info on PCOs can be found here

• Function coverage — have you exercised the features and capabilities?
• Data coverage — have you tested the relevant values, boundaries, combinations?
• Platform coverage — environments, OS versions, hardware, browsers?
• Operations coverage — how users actually interact with the system?
• Code coverage — lines/branches executed (acknowledged, but not privileged over other types)
• Risk coverage — have you addressed the most important risks?

Depending on your chosen API client, will determine the syntax you will need to learn

If you use Bruno or Postman, you will need to be familiar with some basic JavaScript and chai assertions

Assertions

Assertions allow you to declaratively write tests without writing any code.

Scripts

Scripts are any code that runs before or after a request that is not a test. Scripts can be used to populate test data or achieve many other things that aren't tests.

Scripts are categorized as either pre-request or post-request depending on when they run relative to the request being sent

/**
 * This function prints a string to the Postman Console.
 * @param {string} data - The text to print to the Postman Console.
 */
function logger (data) {
    console.log("Logging information to the console")
}

Tests

Tests on clients like Bruno or Postman are usually written in a modified version of JavaScript with a syntax made for each client.

They are written in a BDD chain language known as Chai BDD

pm.test("Status is 200", function () {
  pm.response.to.have.status(200);
});

These tests are separate from Unit tests, integraton tests or contract tests that will usually be written by the devleoper and in the same langauge and framework that the API was developed in.

Developing an API is obviously different to testing it, so I've not covered those tests in this tutorial

Documenting tests

In a lot of projects, documentation about APIs is often limited or non-existent. When I have tested an API I often add documents to the API collection itself whether in Postman or Bruno.

This makes it easier for the next person to look at the collection and gather information they can use for their mental model or use as a kind of oracle.

What automated checks cannot do

Automated checks are mechanistic and cannot look for new problems. That doesn't mean they cannot be used to augment exploration, but by themselves they cannot find new problems.

CLI test runners

Once you have a collection with tests that you want to run consecutively, you can use a CLI runner to run those tests on demand