BINMEN: A Practical Heuristic for API Testing

Boundary, Invalid, Null, Method, Empty, Negative

Created by Gwen Diagram and Ash Winters

When / how to use BINMEN?

BINMEN is a mnemonic heuristic to help with direction when testing an API. When we are exploring an API, there are a lot of avenues we could take. BINMEN is just one avenue to help you explore your API with a bit more focus than randomly clicking around or sending out requests (which can also be valuable or gain information, but has diminishing returns). A mnemonic is a memory tool to help you remember something, so by remembering BINMEN, it should direct you towards boundaries, invalid, etc.

Boundaries

Boundaries are where one thing ends and another thing starts. A limit of an area. Boundaries in software can be explicit or implicit. A boundary can be part of a discussion around a requirement or it can be completely overlooked. Some boundaries are obvious, some boundaries are due to limitations of deeper libraries, software constraints and memory sizes. When trying to simplify boundaries, in standards like ISTQB there is often a bias towards equivalence partitioning and boundary value analysis. You will see testing techniques suggesting go 1 number below a boundary, hit the exact boundary, go one number above the boundary. Although this is looking for problems in the software, it is a very structured and uncreative way of testing. Here are some questions to ask about boundaries that will serve as heuristics to get a bit deeper into the software.

• Ask PO / BA / Developer what boundaries they think exist?
• Ask them why they think those boundaries exist?
• Is the boundary inclusive or exclusive?
• What happens on the exact boundary?
• What happens if the boundary is breached by 1?
• What happens if the boundary is breached by a lot? 10x? 1000x? 100000000x?
• What error handling do we expect around boundaries?
• What boundaries exist within the types that we've used? What is the max number of a 32 bit integer? 64 bit? Items in a list? Chars in a name?
• Are there limits to files being saved? Naming of variables? File names? File sizes? API request payloads? SQL entities?
• How does the system react to extreme data?
• How can a tester follow the data downstream after breaching known boundaries?
• If a tester tests multiple boundaries do they get cascading errors based on the order in which the limit was breached?
• Are the boundaries the same for all users, or do different roles or plans have different limits?

Invalid

Invalid data is data that breaks a rule, whether explicit or implicit, of a system. Invalid data could be that a certain type is expected in a certain field or that a name needs to be of a certain length to be considered a name (I have a friend called Se who gets this a lot). Requirements will also contain lots of rules and there are people (oracles) that can help you understand why a rule is in a certain part of the system. Rules can also be based on things that were never said or are not known. Similar to boundaries, rules can be in systems or libraries up and downstream of the system you're testing.

• If someone was to send the right data in the wrong format, is that a client error or should the API be lenient and coerce it?
• Are special characters accepted?
• Are emails allowed to have + extensions?
• Do phone numbers with international codes get accepted?
• What happens if a tester passes in expired API keys or tokens? How about passing nothing as AUTH? Choosing the wrong auth type?

Null

Null and empty are similar, but you can ask similar questions to cover similar ground although results may be different. For example, an API payload can usually handle the "null" of its intended type, but by removing it entirely, it usually causes schema issues. Some examples of what to consider when null comes to mind:

• URL paths
• Parameters
• Headers
• Values in the payload

Key questions to consider:

• Is there a difference between null and not sending the field at all? Should there be?
• Can a field be intentionally cleared/unset? If so, how is that expressed?

Method

Method (also known as verbs) refers to the API request type such as GET, POST, PUT as used in CRUD. You could ask whether the endpoint is accessible using different verbs. You can also ask if there are any preflight requests. Headers are worth lumping in here too, as the HEAD endpoint can also be valuable. Learn about pre-flight requests and ask if they are required and have been considered.

• What are the risks of being able to access endpoints with a different method?
• Is there a pre-flight request required? How does it look? Any security headers shared with the browser? Is any origin actually being blocked?
• Is any sensitive data exposed in other methods?
• Can someone gather information by passing in methods like HEAD that they shouldn't be seeing?
• Should all methods be documented?
• Should any be hidden?
• Should any methods have feature flags or secret flags for certain environments?

Empty

Similar to nulls, but actual empty objects and arrays rather than 0s, false types and nulls.

• Is empty a valid state or just a mistake?
• What's the difference between "", " ", {}, [], etc?
• Trailing whitespace — how is it handled?
• What errors are thrown when we provide empty values? Empty params? Empty JSON? Empty payload? No payload (in tools like Bruno / Postman)?

Negative

What is negative is usually more context dependent than the other letters. Negative could mean a domain constraint in the database or it could mean going below zero on a field to see a minus value. It can come across as the most shallow of the letters (except method) and limiting, but you can also consider negative testing as part of this heuristic — i.e. look for other problems that haven't been covered yet.

Contextual examples:

• Quantitive: What happens if a customer puts less than 1 item into a shopping cart? Are users presented with a message? Can they checkout with nothing in the shop?
• Percentage: If a percentage is involved, what happens over 100% or below 0?
• What happens if a mandatory field is missed? Does the user receive instructions on how to resolve it?
• What negative values should absolutely never happen in a user journey?

Final thoughts

I use the BINMEN heuristic a lot when exploring an API, and although I really like BINMEN, there are a few pitfalls that I fall into that I'd like to highlight. One of the main ones is that I quickly find scope creep or get error fatigue when looking at things like nulls or negatives. APIs have schemas that need to be respected and of course adding in an empty {} somewhere that it doesn't belong will give some kind of error. The question is whether the system needs to handle every error. A lot of the APIs I've worked on have been very light on error handling, because they are integrating with a very limited system in the first place.

Another trap I see other people fall into is going into the weeds with things like methods. "Should I get a 404 when I change this POST request to a GET? What about HEAD? What about PUT?" Well, maybe a better question is: what are the risks of those endpoints being accessible?

Overall, this is a really good heuristic for learning about an API quickly and can easily lead to gathering very useful information about the state of an API.