In this article, we’ll explore how four AI models—Postbot, GitHub Copilot, ChatGPT and Pulse—are reshaping API testing. These AI systems are not just AI tools for API testing; they’re intelligent assistants capable of understanding code, context, and requirements, and then generating test cases with remarkable accuracy and speed. We’ll take a deep dive into their features and capabilities, comparing how they stack up against each other in the realm of API testing.
API testing, traditionally a time-consuming and labor-intensive process, has undergone a remarkable transformation with the advent of AI. These intelligent systems possess the ability to analyze code, understand context, and generate comprehensive test cases with remarkable accuracy and efficiency, empowering developers to create comprehensive test suites with confidence. Let’s explore how these AI tools for API testing can revolutionise the software development process and what are the advantages and challenges each of them possess.
PostBot
PostBot is the Postman AI assistant tool. It can help you to debug and understand APIs, write tests, create Postman Flows, visualize received responses, and write API and documentation. You can use built-in suggestions to quickly tell Postbot what you need, or be more descriptive in your requests.
Pros:
- Potential for improved test quality:
- It nests assertions within the same test, rather than a separate test for each.
- It labels each test with a useful name
- Quick Iteration and potential for rapid learning:
- AI can refine code generation through minor adjustments to prompts, allowing tailored tests to be quickly generated and adapted. Postbot remembers your requests and allows you to modify the tests as needed.
- Security testing: Postbot streamlines code generation for security testing, reducing the time and effort required to create test scripts for vulnerabilities like local file inclusion (LFI), server-side template injection (SSTI), and command injection.
- Automated code generation: Testers can focus on defining test cases and prompts, while the AI handles the code generation.
- Auto-complete. If you have a response available, type pm.test and it will suggest important tests that you might be looking for. Useful if you want to improve the coverage of your test suite.
- Prompt. You can use the chat to ask for specific or general tasks.
- Efficiency and productivity: By automating initial script creation, Postbot enhances efficiency and productivity. Postbot simplifies the scripting process, making it accessible to testers with varying coding skills, and allowing them to focus on defining test cases and prompts.
- Visualization: Utilize Postbot to generate a Visualizer for effortless viewing of response data in various formats such as tables, charts, or graphs, without the need for manual code writing. Writing the code to configure the visualizer is tricky. However, the auto prompts work well. When selecting the Visualize Response as a table, Postbot provides the necessary code and displays all the properties, making this feature genuinely helpful.
Cons:
- It’s important to be cautious when using this AI tool for API testing as it may generate incorrect syntax at times. For instance, if the cursor is positioned within an existing test when querying Postbot, the generated syntax may not align accurately.
- It overlooks existing tests; thus, if a test verifies a response status code of 200 and you ask it for more tests, it may generate duplicate tests.
- Postbot may not consistently provide accurate responses, particularly when queried about variable scopes. Its primary focus appears to be on executing actions rather than addressing documentation-related inquiries.
- Sometimes, the AI might not perfectly grasp your intentions and generate the exact scenario you’re imagining. In these situations, before diving into coding, you can ask Postbot to create a custom test for you. It’s worth noting that Postbot is quite adept at understanding complex prompts.
ChatGPT
Refer to the tool described on the page “AI For Testing: A Comparison of Unit Testing Tools”.
ChatGPT is a language model announced by OpenAI on November 30, 2022. While the primary purpose of an AI chatbot is to imitate human conversation, ChatGPT is highly versatile and can perform a wide range of tasks such as coding and debugging software, providing responses to questions, and more.
Pros:
- ChatGPT can generate code snippets that demonstrate how to interact with specific APIs, streamlining the integration process and making it more efficient.
- ChatGPT can also help produce test scripts and frameworks such as Rest Assured or Postman.
Cons:
- You have to write the requirements to generate test scenarios and then ask it to generate the code. In the same way, you can ask it to write test cases for the API by sharing the API documentation, but you will have to ask it again to transform them into code. It may be necessary to iterate over the scenarios to get the desired outcome. When using ChatGPT with a private API, be aware of potential security risks.
- The output provided by ChatGPT may not be comprehensive or relevant and may not be able to generate edge tests or cases for corner scenarios that are critical for software testing.
- ChatGPT cannot recognize the context or purpose of software applications, which may result in inaccurate responses.
GitHub Copilot
GitHub Copilot is a tool that can assist in code generation and suggest code snippets based on comments and descriptions in your code. However, it does not automatically create unit tests or guarantee the quality of your tests.
Pros:
- The structure of the code and the code itself are generally good.
- Suggestions are very quick. A good choice of suggestions, all feasible.
- The assertions, although not necessarily all correct, are very plausible suggestions.
- Once you start seeding the first test, Github co-pilot can provide some examples with a minimum amount of effort.
- Copilot knows popular API frameworks and how to use them.
- Provides updated code to resolve breaking tests.
Cons:
- To generate test scenarios, you will need to provide the necessary context. It may be necessary to iterate the process to get the desired outcome.
- Not precise. It might not consistently offer the most efficient solution. Therefore, it’s crucial to review the suggestions and adjust them as needed. This may involve altering the generated code or writing additional code to accomplish the desired outcome.
- Users still need a solid understanding of programming concepts, syntax, and best practices to evaluate and adapt the suggestions appropriately for their specific needs.
Pulse from Testifi
Pulse is an AI-powered API testing tool that can automatically generate tests, execute them, and analyze the results to identify and report errors. It can test any type of API, including REST APIs, SOAP APIs, and GraphQL APIs. Pulse analyzes the API documentation, generates API tests according to the details in the documentation and best testing practices, and analyzes and presents the results in an easily consumable manner. No coding or implementation knowledge is required. The schema URL/file, server URL, and authentication details are enough to start testing with Pulse. Additionally, it can automate end-to-end, unit, integration, functional, and UI/UX testing.
Pros:
- Enhanced Test Coverage: Pulse analyzes all tests, eliminating redundant ones and presenting those that represent both intended functionalities and unintended implementations/bugs in a convenient dashboard view. This allows easy access to results and ensures comprehensive coverage.
- Automated Test Generation: Pulse can automatically generate tests covering various scenarios, including edge cases and negative tests. Users have the option to upload test data for endpoints, though only positive examples are necessary for successful response generation; Pulse handles negative examples by default. Alternatively, users can add values as examples in the schema.
- Improved Security: Pulse identifies and reports security vulnerabilities in APIs, enhancing protection against potential attacks.
- Jira Integration: Pulse can automatically generate test cases in Jira, with planned integration for Xray in future releases.
- OpenAPI Documentation Compatibility: Note that Pulse may not generate as comprehensive or accurate tests if your API documentation is not in OpenAPI format.
Cons:
- Pulse tries to generate suitable data for each parameter, yet achieving full coverage may need specific values for certain endpoints.
- As a paid tool, Pulse’s cost may pose a barrier for some organizations.
- False Positives: Pulse may occasionally produce false positive results.
- Integration Limitations: Pulse does not integrate with all CI/CD tools.
- Overall, the comprehensiveness and structure of your API documentation significantly influence Pulse’s test generation capabilities. Nevertheless, Pulse remains a valuable tool even in instances where API documentation is not in OpenAPI format.
AI for Testing – The Zartis Approach
At Zartis, we understand the immense potential of AI and offer comprehensive services to help businesses unlock its transformative power. We are dedicated to partnering with organisations to develop and implement customised AI strategies that address their unique challenges and fuel exponential growth.
Whether you aim to boost operational efficiency and productivity, enhance customer experience, or unlock new revenue streams, Zartis has the expertise and solutions to achieve your goals in this field. We leverage the latest advancements in AI technologies, such as machine learning, natural language processing, and computer vision, to create innovative solutions that deliver measurable business outcomes.
Don’t miss the opportunity to unlock the immense potential of Artificial Intelligence and propel your QA teams forward. Contact Zartis for AI Consulting Services today to schedule your free consultation and embark on a journey of transformative growth. Let our experts guide you towards a QA strategy powered by AI, where innovation fuels success.
Author:
Carolina Bonet is a passionate senior QA Engineer. She has a strong experience in test automation working with various languages including C#, TS and JS, and frameworks such as K6, Selenium and Cypress. She has actively contributed to the improvement of QA strategies, optimizing company processes for enhanced product quality. Carolina is known for her passion for learning and adapting to new technologies.
