API Testing Showdown. Postman vs Pytest. Part 5

Published in

Exness Tech Blog

12 min readNov 25, 2024

The 5th part concerns Postman’s helpful features unrelated to auto-testing and how we can combine Postman and pytest to get more.

The previous parts of this comparison are here: part 1, part 2, part 3, part 4.

There are some neat but out-of-scope features we need to talk about…

As you remember, in this series of articles, we’re talking about API testing automation.

However, we need to discuss some other Postman features indirectly related to API auto-testing. The picture given here would only be complete with them.

Specifications import

Postman

The quickest way to kickstart your testing is to import your specifications into Postman. Since you can import a specification in several ways, let’s try OpenAPI by URL. Then, Postman automatically generates requests and examples where possible.

This is just a set of requests with no connection between them. But starting with a set of requests rather than a blank collection is much easier.

Pytest

Pytest is a general-purpose framework that “doesn’t know” what a request is. We use it coupled with the requests library. It’s impossible to get a set of requests in a few clicks. On the other hand, this is different from how Pytest functions. It operates tests, not requests.

I cannot but agree that Postman wins; it is easier to start testing with.

However, I also want to mention that generating clients (wrappers) around Python’s request library is possible. As we’re stepping into a wild field of framework creation, I’ll be short. The standard solution for client generation is the OpenAPI Generator. You may need time to make it work, but the results are worth it. If you’re already working with pytest, your src folder contains clients, models, and helpers. An OpenAPI generator is exceptionally flexible, and being configured once, it can save plenty of time by cutting tedious work. This is not only valid for the beginning of your project; clients can be automatically updated with specs.

Why is it worth mentioning? To some extent, we can draw a parallel between Postman requests and clients’ methods, representing endpoints. Both are used in tests and end up the same—a server gets called via HTTP.

Writing tests with AI helper

Postman

We’re on an AI hype train, and one of the fresh Postman features is an AI assistant AKA Postbot (beta). It can help you write tests like code snippets before. However, AI cannot help you support tests and cannot create complicated flows with multiple requests.

What is the Postbot capable of?

Tests generation

First of all, we can generate tests (collection).

Postbot won’t generate additional tests if you’re trying to generate tests for a whole collection and your request already has one.

However, you can complement your existing tests with AI-generated ones if you call the bot directly from the code editor.

It analyzes existing code poorly, as the tests may reappear:

As for the content of generated tests, Postbot performs a general analysis of the response.

Sadly, the bot cannot connect GET following the POST to check data consistency. You still have to write such tests by yourself.

In the CRUD example above, the ID created in the POST is passed to the GET response via the collection variable. I implemented this; the bot cannot determine how these requests can be related.

I also tried to generate tests for an OpenAPI-based API. It is not much better, but at least it tried to verify the length of the content.

As we have CRUD here, comparing against constant 6 is not a proper test; it needs tweaking, but I’m happy to see it here.

And, of course, it cannot analyze specifications deeply and create new requests; it can only write tests for existing ones. You cannot automatically cover business flows; your capabilities are too limited. I find it useful, though, as Postman is a tool that wants you to spread copy-paste all over the workspace. This generator of tests can ease your work.

Collections: Postbot demo (broken flow), Postbot demo (working flow), Postbot demo (Flasgger RESTful).

Fixing your tests

The bot can help you to fix your tests. I tried breaking the standard “Status code is 200” test. Incorrect status code (“20” instead of 200), test name (“Status code is 201”), and broken object name (“pm.resonse” instead of “pm.response”) have been fixed properly.

Oddly, it automatically translates comments into English.

Before:

After:

Collection

Visualization

Visualizing responses in Postman is a pain in the neck. I can hardly imagine a real user manually setting up all this code. Fortunately, Postbot allows you to generate code automatically via requests like “Visualize response as bar graph”…

…or “like table”

There is a high chance you’ll need to fine-tune the data relationships. But it is much better than creating this code from scratch.

Collection

What about telling a joke?

My curiosity led me to try something completely different. Yes, the Postbot can tell you a joke. Unfortunately, it messes up the visualization template.

This joke functionality looks more like an easter egg. If I try something more complicated, it won’t answer the question and work like the “Generate tests” request.

As AI is developing so fast nowadays, we will probably see real AI-powered auto-testers in the upcoming years 🤞. As for now, Postman AI is nothing but a little helper for inexperienced engineers. It doesn’t look like something powerful enough to ease any serious testing.

Pytest

Pytest is a testing framework that doesn’t offer an AI test generator. But we’re in a world of Python, right? So we can use other AI tools, much more powerful than Postbot. Let’s ask notorious OpenAI v3.5 to generate some pytest API tests. For this demo, I’m using OpenAPI-based API again.

The request I used looks like this:

Write API auto-tests on pytest framework for the http://flasgger.pythonanywhere.com/restful/apidocs/#/

The answer was amazing. It explained how to install pytest and requests and generated several smoke tests after that. And here I see more advanced tests. ChatGPT not only created simple CRUD tests, one per “C”, “R”, “U”, and “D”, but it has also figured out how to add and then check the added data.

# Test: GET request to retrieve a specific item
def test_get_item():
 # First, add an item
 add_response = requests.post(f"{BASE_URL}/items", json=sample_data)
 item_id = add_response.json()["id"]

 # Then, retrieve the added item
 response = requests.get(f"{BASE_URL}/items/{item_id}")
 assert response.status_code == 200
 assert response.json()["name"] == sample_data["name"]

# Test: DELETE request to remove an item
def test_delete_item():
 # First, add an item
 add_response = requests.post(f"{BASE_URL}/items", json=sample_data)
 item_id = add_response.json()["id"]

 # Then, delete the added item
 response = requests.delete(f"{BASE_URL}/items/{item_id}")
 assert response.status_code == 200
 assert response.json()["message"] == "Item deleted"

Remember that you must write tests manually to create add+get and add+delete tests in Postman, as Postbot can only create general tests.

And these comments weren’t added by me!

The full conversation can be found here.

Even though ChatGPT is not a pytest feature, using even free additional tools can easily yield better results than Postman AI can offer.

The bad news is that AI still cannot do the full job, even in Python. You will need a test framework and more complicated tests to cover more advanced flows. But the way technology evolves is amazing!

Postman Interceptor

One of the most prominent features available in Postman is Interceptor. If you’re testing a backend-for-frontend and have a frontend available, you can add the interceptor plugin to your Google Chrome and collect all the calls your frontend makes. It’s just like Selenium IDE but for the backend.

After collecting data, you can select required requests in Postman to move them into a collection.

An advantage over specs import is obvious: you capture business flows right as they are implemented, which is the easiest way. The difficulty is that you capture a lot of garbage and useful requests, so separating the wheat from the chaff takes some effort.

As in the case of Selenium IDE, flows captured with little or no post-processing are hard to maintain and often require re-recording. However, the Interceptor may be a good starting point.

API security and governance rulesets

Postman allows you to set security rules right on the level of API definitions (the same for governance rules). It is not about functional testing or testing in principle, but it can be a good add-on for establishing quality. These rules are applied early, perfectly fitting the trendy “shift left” principle.

Unfortunately, configurable security rules are only available for enterprise subscriptions.

Another con is that the rules are applied against API definitions stored inside the Postman platform. So, this neat feature is for teams building the whole work around this platform.

Postman as a platform

While reading this section, you’ve probably noticed that the most interesting Postman features are

not related directly to testing.
tightly coupled with each other (i.e., Postman platform).

The only way to take everything from Postman is to use it as a platform for your whole team. This is how they build business and how they get financed. An API-first team that tamed Postman can get a lot of benefits in terms of quality assurance right out of the box:

clean and easier-to-access documentation
auto-generated mocks, establishing easier early testing
quick (but dirty) requests and test generation
rulesets applied in the early stages of API development
monitoring is integrated too, you can quickly start running suites against nightly builds
integrated environments
visualizations
etc.

Sadly, even if you have a 100% integrated team and paid Postman and the whole development process goes through the Postman platform, you’ll still face the fact that Postman is imperfect at API testing automation. The hard truth is that nothing more complicated than the “GET /hello_world” app can be tested thoroughly via Postman without stretching TTM to eternity.

But you don’t have to! You can use pytest and other full-fledged testing frameworks.

Can I get the best of both worlds?

So, if Postman is such an incredible platform that’s so bad for auto-testing, maybe we can combine them? Yes, there are potential ways to do that.

Interceptor/Import -> proto-tests -> pytest

If you already have a frontend and want to create a set of backend smoke tests based on user flows, you can easily sketch it using the Interceptor. After collecting and sifting requests, you can continue with writing JS tests. Thus, you can quickly gain minimal coverage with minimal effort. After that, you can rerun tests in CI/CD using Newman or schedule nightly runs via collection runner or monitor. Once you need more advanced testing (which will happen sooner rather than later), you can convert your requests to pytest scripts and start creating test frameworks while understanding how the system under test operates.

Exploratory testing in Postman, auto-testing in pytest

Most testers use Postman as an HTTP client. And this is understandable. This is a user-friendly GUI-based tool that makes request sketching easy and fun. Thus, it is one of the best solutions for hypothesis checking and exploratory testing. Again, after gathering info during an exploratory testing session, you can convert your collections to Python scripts and proceed with automatization via pytest.

Security rules in Postman, functional tests in pytest

Auto-checking of security/regulatory rules is a good add-on to usual functional testing. This is a solid example of the early testing principle, as the result can be seen right after a change happens. If your company is subscribed to Postman, it’s worth trying.

Dev/Int testing with pytest, SIT/UAT with Postman Flows

Postman Flows is a too raw project, but I believe it might be more useful in the future. It is unlikely that it will supersede something like pytest but it can help you with the late stages of a release cycle, like SIT or UAT.

Postman Flows can be useful for SIT as they push you to concentrate efforts on calls and data transfer between requests rather than response verification.

This tool can be helpful for UAT in a specific way. Imagine you’re a business owner in a company heavily relying on outsourcing. You have a small testing team of mostly manual testers or a group of business users that run acceptance testing. Postman Flows can greatly help both categories as it has a low entry threshold and lets you cover certain business processes.

The conclusion

I feel constrained when I work with Postman. Postman requests are easy to create but hard to use in auto-tests. Trial of creating a more or less big auto-suite will almost always end up as a mess with a lot of copy-pasted and hard-to-maintain code. Workarounds are just not convenient enough. Even if I talk about combining Postman and Pytest, you can see that I still leave the most significant part of auto-testing to Pytest. Pytest just does auto-testing better.

Materials

My code

[Demo REST API I used for several occasions [GitHub)(https://github.com/CaptainKoffski/BowserAPI-Tests) [Pytest tests code [GitHub)(https://github.com/CaptainKoffski/postman-vs-pytest) Postman workspace