Software Development Process 11 | Software Verification and Testings

Series: Software Development Process

Software Development Process 11 | Software Verification and Testings

  1. Software Verification

(1) Data of Buggy Software

In fact, a federal report from a few years ago assessed that software bugs are costing the US economy $60 billion every year. In addition, studies have shown that software contains on average 1~5 bugs every 1000 lines of code. So, building 100% correct mass-market software is just impossible. This means that we have to verify the software as much as possible and we will discuss different alternative ways of very fine software systems with particular attention to the most common type of verification, which is software testing.

(2) The Definition of Failure

A failure is an observable incorrect behavior of the software. It is conceptually related to the behavior of the program, rather than its code.

(3) The Definition of Fault

A fault (aka. bug) is an incorrect piece of code. In other words, a fault is related to the code and is a necessary, but not sufficient

(4) The Definition of Error

An error is the cause of a fault. It is usually a human error, which can be conceptual, for example, a typo or something along those lines.

(5) Software Verification Approaches

There are several ways to verify a software system. Among those, we will discuss four mainstream approaches.

  • Testing (aka. Dynamic Verification): Software testing is the most popular and most used approach in the industry.
  • Static Verification
  • Inspections
  • Formal Proofs of Correctness

(6) Software Testing (aka. Dynamic Verification)

Testing a system means exercising the system to try to make it fail. More precisely, let’s consider a program. The input domain is the set of all the possible inputs for the program, and its output domain is a set of all the possible corresponding outputs. Given this context, we can define what a test case is.

A test case is a pair that consists of input i from the input domain D, and then, expected output o from the output domain O. And o is the element in the output domain O that a correct software would produce when run against i.

Test Case: {i ∈ D, o ∈ O}

We can also define the concept of the test suite, which is a set of test cases.

The benefit of this approach is that,

  • No false positives: when testing generates a failure, that means that there is an actual problem in the code.

The downside is that,

  • Highly incomplete: Even in the best scenario, testing can consider only a tiny fraction of the problem domain, and therefore a tiny fraction of the program’s behavior.

(7) Static Verification

Static verification tries to identify specific classes of problems in the program, such as null pointer dereferences. Unlike testing, what it does is that it does not just consider individual inputs, it instead considers all possible inputs for the program. So it considers in a sense all possible executions of the program and all possible behaviors of the program, that’s why we say that verification is complete.

The benefit of this approach is that,

  • Considers all program behaviors

The downside is that,

  • Generate false positives: Due to the limitation of this kind of analysis and due to infeasibility issues, static verification considers not only all the possible behaviors but also some impossible behaviors. For example, it might report a possible NULL pointer to the reference that cannot actually occur in practice.

(8) Inspections

Inspections are also called reviews or walkthroughs. And unlike the previous techniques, inspections are a human-intensive activity. More precisely, they are a manual and group activity in which several people from the organization that developed the software, look at the code or other artifacts developed during the software production and try to identify defects in these artifacts. Interestingly, inspections have been shown to be quite effective in practice and that’s the reason why they’re used quite widely in the industry.

The benefit of this approach is that,

  • Systematic and thorough: Inspections are done in a rigorous way.

The downside is that,

  • Informal and subjective: They are nevertheless a manual process or a human process. The result may depend on the specific people who perform the inspection.

(9) Formal Proof

Given a software specification or a formal specification, so a document that formally defines and specifies the expected behavior of the program. A formal proof of correctness proves that the program is verified and actually implements the program specification. It does that through a sophisticated mathematical analysis of the specifications and of the code.

The benefit of this approach is that,

  • Provide strong guarantees: They can guarantee that the program is correct, which is not something that any of the other approaches can do, including static verification.

But the main limitation of formal proofs is that,

  • Complex and expensive: they need a form of specification or a complete mathematical description of the expected behavior of the whole program. Unfortunately, such a specification is rarely available, and it is very complex to build one. In addition, it is also very complex, and possibly expensive, to prove that the program corresponds to a specification. That is a process that requires strong mathematical skills and, therefore, a very specialized personnel.

2. Testing

(1) The Definition of Testing

As we have mentioned earlier, testing means executing the program on a tiny sample of the input domain, which is all the possible input data. There are two important aspects of testing,

  • Dynamic Technique: the program must be executed in order to perform testing
  • Optimistic Approximation: the program under test is exercised with a very small subset of all the possible inputs as we just said. And this is done under the assumption that the behavior with any other input is consistent with the behavior shown for the selected subset of input data. And that is why it is an optimistic approach.

(2) Testing Granularity Levels

There are several granularity levels for the developer’s testing, which means the tests are performed within the testing organization, or by somebody who is doing like third-party testers on behalf of the testing organizations. For example,

  • Unit Testing: the testing of the individual units or modules in isolation.
  • Integration Testing: Integration testing is the testing of the interactions among different modules and it can be performed according to different strategies depending on the order in which the modules are integrated and on whether we integrate one module at a time or multiple modules together, all at once.
  • System Testing: System testing is the testing of the complete system and it includes both functional and non-functional testing. System testing includes both functional and non-functional testing.
  • Functional Testing: Functional tests are the test that aims to verify the functionality provided by the system.
  • Non-functional Testing: Conversely, non-functional testings are the ones that target, surprisingly, no functional properties of the system. For example, the non-functional tests will include performance tests, load tests, robustness tests. The aim is to assess different qualities of the system such as reliability, maintainability, usability, etc.
  • Acceptance Testing: The validation of the software against the customer requirements. So this is the testing that makes sure that the system does what the customer wants it to do.
  • Regression Testing: Regression testing is the type of testing or retesting, that we perform every time that we change our system and we need to make sure that the changes behave as intended and that the unchanged code is not negatively affected by the modifications. The regression errors are parts of the code that aren’t related to the changes, are actually affected by the changes, and start misbehaving.

(3) Alpha Testing Vs. Beta Testing

There are two other kinds of testing that are also related to testing phases, and these are alpha and beta testing.

  • Alpha testing is the testing performed by distributing a software system ready to be released to a set of users that are internal to the organization that developed the software. So you can consider these users as guinea pigs that will use an early version of the code and will likely discover errors that escaped testing and will have made it to the field if not caught.
  • Beta testing is the next step after alpha testing, in which the software is released to a selected subset of users outside your organization. And also, in this case, the users are likely to discover latent errors in the code before it is officially released to the broader user population, so before we have an actual product release.

The reason why we conduct alpha testings rather a one-step beta testing is that alpha testing is performed to iron out the very obvious issues that still escape testing. And the rationale is that alpha testers have a higher tolerance for problems than beta testers, who expect a mostly working system.

(4) Black Box Testing Vs. White Box Testing

Now let’s see an overview of the black-box testing and the white-box testing, and we will talk more about them in the following series.

  • Black-Box Testing: Instead of looking inside the software, we just going to look at the description of the software. So this is the testing that is based on a description of the software, which is what we normally call the specification for the software. And what black box testing tries to do is to cover as much specified behavior as possible, and the main limitation of black box testing and the reason why this is complimentary to white-box testing is that it cannot reveal errors due to implementation details.
  • White-Box Testing: Conversely, white-box testing looks at the codeand uses this information to perform the testing. So white-box testing is based on the code, its goal is to cover as much coded behavior in this case, as possible, and its limitation is that unlike black-box testing, it can’t reveal errors due to missing paths.