The classic dilemma for testers - Robot or pytest

For any organization, customer confidence is of the highest priority, and a lot of it is dependent on reliability which is mostly dependent on the quality assured by the product and service you are providing. This has increased the importance of QA, Testing teams and the framework you are using for testing. Choosing the right test framework for a project is a critical decision. As of today, there are many open source as well as proprietary test automation frameworks in the market which you can consider depending on the project's requirements. Through a series of blogs, with this being the first part, I will attempt to explore the criterion for selecting such frameworks and evaluate some of the best ones.

With Python’s increasing advent in the data science and machine learning domains, it is often confusing even for experienced professionals to pre-decide on their approach to testing for agile projects. With Robot's keyword-driven approach, which makes it user-friendly, and pytest being easily maintainable for continuous changes, it is but natural that these frameworks continue to gain the most popularity.

The Robot framework can be thought of as a test automation template that can be used to automate scenarios with a keyword-driven approach. The keyword-driven approach simplifies the task of writing the test cases. The user just needs to import the specific libraries (default as well as custom) for using the inbuilt keywords. This makes the code extremely easy to read and understand as well. It can be used to automate complex scenarios and, at the same time, offers the flexibility to use languages like Java and Python.

The pytest framework, however, is specifically designed to simplify and enhance the unit testing experience. pytest proves extremely handy for unit testing scenarios wherein the test code block remains almost similar while only the inputs vary. Built-in tags like ‘parameterize’, and ‘fixtures’ help in avoiding repetitive code and make the code more maintainable, but only at the cost of reduced readability. Due to their inherent capabilities, Robot proves itself more useful while testing the entire stack, whereas pytest is limited only to unit testing. Let’s try and get a deeper perspective of both of them through a set of examples:

Usage:

Being a data-driven test automation framework, Robot needs the user to provide the test data through a config file. A simple example using this approach for logging into an application has been shared below:

*** Settings ***
Test Template         Login with invalid credentials should fail
*** Test Cases *** USERNAME PASSWORD
Invalid Username invalid ${VALID PASSWORD}
Invalid Password ${VALID USERNAME} invalid
Invalid Both invalid invalid
Empty Username ${EMPTY} ${VALID PASSWORD}
Empty Password ${VALID USERNAME} ${EMPTY}
Empty Both ${EMPTY} ${EMPTY}
*** Keywords ***
Login with invalid credentials should fail
[Arguments] ${username} ${password}
Input Username ${username}
Input Password ${password}
Submit Credentials
Error Page Should Be Open

In the above example, “Input Username, Input Password, Submit Credentials, Error page Should be Open” are nothing but keywords that indeed refer to a Python library function. The Keywords are composable. This means the user can define new keywords(custom) which use existing keywords. This way, the user can abstract details of testing to something that makes immediate sense. For example, we don't need to know what exactly the “Push buttons” Credential actually does unless we want to. Test cases are, therefore, clear and readable, with just the right level of abstraction to convey the intent of the test.

On the other hand, pytest uses fixtures/parameterization to build functions/test cases. Parameterization allows users to define a function that takes multiple arguments. This proves advantageous with each of it constituting a single test case as opposed to Robot framework where the user needs to write a separate test case every time even though just the inputs are different. A simple example of this is:

Example:

@pytest.mark.parametrize("username, password, expected_result", [
(“invalid_username”,”valid_password”, “Login failed”),
(“valid_username",”invalid_password”, “Login failed”),
("valid_username", “”, “Login failed”),
])
def test_eval(username, password, expected_result):
  Login_result = login(username,password)
   assert  Login_result == expected_result

The above example shows how a single test_eval can execute 3 test cases using parameterize tag. Similar to parameterize, we can also pass arguments using fixtures. Fixtures are very handy when you want to artificially separate out the test cases without having to move any data from one file to another by defining the scope of execution.

Example:

@pytest.fixture
def test(username,password):
  a = username
  b = password
  obj = cal(a,b)
  return obj
 def test_wrong_username(test):
  response = login(test.a,test.b)
  assert response == “Login failed” 
def test_wrong_password(test):
  response = login(test.a,test.b)
  assert response == “Login failed” 
def test_empty_password(test):
   response = login(test.a,test.b)
   assert response == “Login failed” 

The above example shows how a single fixture is shared for fetching the data inputs to perform login operations with the scope of the fixture being just the scenario under execution.

Reporting:

The Robot framework gives extremely presentable test reports while offering the ability to the user to organize the tests with a very simple and robust tagging system. Robot has a test runner that can provide output not only in XML but also in multiple other formats using its listener interface.

On the other hand, pytest’s built-in reporting mechanism does not provide a full-fledged report, and the user may have to get some external plugins to get a more detailed report.

Some more fundamental differences between the Robot and pytest automation framework are:

Modular Design:

By using modular fixtures, it is easy to modify or add assertions because test cases are reduced in the pytest functions as compared to Robot framework. By taking advantage of pytest.mark.parametrize decorator, adding an extra scenario is just a matter of adding the additional input tuple.

Custom Asserts:

pytest allows the users to use the standard Python assert for verifying expectations and values in Python tests, or the users can implement their own assertion using the following:

pytest_assertrepr_compare hook.

Execution Time:

With pytest, execution time is reduced drastically (35 to 40%) as compared to Robot Framework test-suite.

Plugins:

Another advantage of switching to pytest is the no. of readily available plugins and the provision to design your own plugins. In Robot, you won’t find the plugins readily available. The best example to illustrate this is Cookiecutter.
Simply install Cookiecutter and generate a new pytest plugin project:

$ pip install cookiecutter

$ cookiecutter https://github.com/pytest-dev/cookiecutter-pytest-plugin

... and you are done. The task of Cookiecutter is to simply create a task project from task template.

Test data organization/maintainability:

Robot framework is an efficient way to organize the test cases. It is very handy with the manner in which we can write the test cases. It increases code readability. On the other hand, pytest is an effective way of clubbing repetitive test cases into a single test function and passing the variables using fixtures/parameterization. If we compare code maintainability, pytest is better than Robot, since it eliminates the compulsion to use config file for each suite. All the configurations/parameters are present in the pytest file itself as part of the main code, and thus user overhead to maintain clumps of config files is nullified.

Both these frameworks are open source as of today and offer great flexibility for test automation depending on the focal points of the project’s testing effort. So, in case you are still doubtful and undecided on either of them, you may want to wait till the next blog in this series is out, wherein I will focus on actually working with these frameworks.