Test Driven Development In Data Science

Ojash Shrestha
5 min readJul 18, 2021

--

In this article, we’ll discuss Test-Driven Development in Data Science. The nitty-gritty details of the need and benefits of Testing are discussed in birds-eye view for Data Science tasks. We’ll learn about Unit tests, their pros, and cons, and some great tools for the testing are introduced.

Testing

Testing is a part of the software development life cycle that has been discussed in the previous article Software Development Life Cycle (SDLC). It is crucial that the code is tested before being deployed. This enables finding errors and faults before they can cause any major impact.

Software testing refers to the process of verification and evaluation that software performs the specific way it was desired to. This can be done by numerous types of testing which provides benefits of improving the performance of the system, reduce developmental costs and prevent bugs.

Why should we perform testing in Data Science?

During the process of Software Engineering and Data Science, there could be problems that would arise that are difficult to detect such as data that were encoded incorrectly, unexpected break assumptions, inappropriate use of features, and more. In order to catch these errors, quality check of code is not enough. A proper check of the accuracy of the analysis needs also be done. Thus, proper testing would help reduce unexpected flaws and better confidence in our results could be achieved.

Unit Testing and Test-Driven Development (TDD) are some of the widely used ways of Testing procedures adopted prevalently for Data Science.

Unit Test

Unit Test is one of the types of different types of testing which is responsible for a specific “unit” of code. This approach is used to test the performance of a single function independently disregard of the other part of the system.

It is essential to note that the tests for the function should be performed in a way it is repeatable and automated. The tests are run for all multiple functions in a program and the system will show if there is any failure for any specific unit and the ones which are successful through the test. More so ever, there are numerous tools that are available in Python for creating effective unit tests.

Pros and Cons of Unit Testing

As the name signifies, unit testing is performed dedicated to the smallest unit of the program, often the function. This makes it isolated from the program and hence no dependencies come into play. Databases, APIs, and any other external sources of information are not required thus, proving a huge security advantage.

However, since the unit test also only tests the units, it isn’t quite confident if the system can function in a holistic manner from the macro point of view in the amalgamation of all the different components. Each unit working properly without fault and the entire system working together perfectly is a different thing. Thus, to solve this Integration Testing would be essential. We could talk about this in detail in a future article.

Some of the Powerful Unit Test Tools

PyTest

Pytest is ranked №1 for the best Python Unit Testing Frameworks. It is a mature testing tool that caters to the entire features of the Python Programming Language and is well suited for most Projects developed in Python.

Unitest

Unitest is ranked №3 for Python Unit Testing Frameworks and includes a built-in standard library with the unitest module to write the tests.

Test-Driven Development (TDD)

Test-Driven Development as the name suggests is the process of software development in which the development is driven by test ie. Tests are written for tasks before the codes are implemented.

The process of TDD involves writing tests before the code and as the test fails initially and later succeeds, we’d know that the task will be correctly implemented. Various edge cases and multitudes of scenarios can be tested even before writing a single function. And as the functions are implemented, the test can be executed and feedback can be received thus supporting us with the analysis to tweak our function code. Moreover, as we refactor our code, the tests would assure that the function is working soundly without us having to worry if the system breaks as the minute part of the system — the function’s working is intact.

Besides, the TDD for Data Science is relatively new and is undergoing experimentation and breakthrough on a daily basis.

Learn more about testing through this video on Testing from Web Apps from the Code Quality & Performance Virtual Conference,

What types of errors does Test-Driven Development enable to solve?

Error of Implementation

The error of Implementation refers to the basic kinds of error that arise due to failure in proper implementation. This could be a basic error in the use of mathematical operations or functions to failure in the control of numerical error accumulation. Such error is prevalent throughout earlier times which has caused huge losses such as during the Gulf War, an American Patriot Missile battery in Dharan, in that Saudi Arabia, failed to track and intercept an incoming Iraqi Scud missile resulting in over 100 casualties; the incident today known as The Patriot Missile Failure.

Error of Interpretation

The error of Interpretation refers to the misinterpretations and misunderstandings that don’t let us question our precarious view of reality, and with the assumption that our values are accurate, we draw conclusions that are fallacious at best. Such instances can be seen and understood when Google announced its Flu Prediction system known as Google Flu that actually couldn’t detect Swine Flu that shook the modern world just some months after Google had announced their system. Similarly, the Professors of the University of Colorado had made a prediction system that had modeled to correctly predict every election since 1980 but failed despite their extremely confident announcement which couldn’t foresee the re-election of Obama in the 2012 Presidential Election in the USA.

Read the full article on C# Corner:

--

--

Ojash Shrestha
Ojash Shrestha

Written by Ojash Shrestha

Man on a Mission - to create epochal impact

No responses yet