Klara is a static analysis tools to automatic generate test case, based on SMT (z3) solver, with a powerful ast level inference system.

Overview

Klara

Klara is a static analysis tools to automatic generate test case, based on SMT (z3) solver, with a powerful ast level inference system. Klara will take python file as input and generate corresponding test file in pytest format, that attempt to cover all return values. For example, following function in file test.py

def triangle(x: int, y: int, z: int) -> str:
    if x == y == z:
        return "Equilateral triangle"
    elif x == y or y == z or x == z:
        return "Isosceles triangle"
    else:
        return "Scalene triangle"

will generate

import test
def test_triangle_0():
    assert test.triangle(0, 0, 0) == 'Equilateral triangle'
    assert test.triangle(0, 0, 1) == 'Isosceles triangle'
    assert test.triangle(2, 0, 1) == 'Scalene triangle'

See the Klara's documentation at https://klara-py.readthedocs.io

Installing

Klara can be installed via pip tool by using:

pip install klara

Usage

We can invoke klara on any python source file, and it will generate a corresponding pytest test file.

$ cat source.py
def foo(x: int, y: int, z: str):
    if x + y > 2:
        return x + y + 12
    elif x < y:
        return x + y
    elif (z + "me") == "some":
        return z + "thing"
    else:
        return x - y

$ klara source.py
$ cat test_source.py
import contract_test


def test_foo_0():
    assert contract_test.foo(0, 3, \'\') == 15
    assert contract_test.foo(0, 1, \'\') == 1
    assert contract_test.foo(0, 0, \'so\') == \'sothing\'
    assert contract_test.foo(0, 0, \'\') == 0

Consult the quick start manual for more examples and guidance. To use it as a static analysis library, go to Inference.

Why Klara?

Klara works on ast level and it doesn't execute user code in any way, which is a very important difference compared to similar tool like Crosshair and Pynguin that utilize concolic symbolic execution that required user code execution that might cause unwanted side effects. Klara work on ast level, combine with data flow analysis that utilize Control Flow Graph(CFG), Static Single Assignment(SSA), use-def chain, etc... to build a powerful python inference system that leverages Z3-solver for constraints solving and path feasibility check. Because of this, Klara is able to operate on both python2/3 source code with the help of typed_ast. To specify the source code is in python 2, pass in -py 2 argument. It's python 3 by default.

Klara can also be used as a static analysis tool, allow user to define custom rule to identify programming bugs, error or enforcing coding standard. With SMT solver support, analysis will be more accurate and greatly reduce false-positive case. For example

4: if v1 < 3: z = 1 else: z = 2 else: z = 3 s = z """) with klara.MANAGER.initialize_z3_var_from_func(tree.body[0]): print(list(tree.body[0].body[-1].value.infer())) ">
import klara
tree = klara.parse("""
    def foo(v1: int):
        if v1 > 4:
            if v1 < 3:
                z = 1
            else:
                z = 2
        else:
            z = 3
        s = z
""")
with klara.MANAGER.initialize_z3_var_from_func(tree.body[0]):
    print(list(tree.body[0].body[-1].value.infer()))

Will print out:

[2, 3]

Because z = 1 is not possible due to v1 > 4 and v1 < 3 is unsatisfiable

The inference system architecture and api is largely inspired by Astroid, a static inference library used by Pylint.

Klara utilize the inference system to generate test case, in other words, it generate test case for all possible return values of the function, instead of generate test case for all control path of the function.

To illustrate the point, consider the function below, with divide by zero vulnerabilities at line 3

def foo(v1: int, v2: float):
    if v1 > 10000:
        s = v1 / 0  # unused statement
    if v1 > v2:
        s = v1
    else:
        s = v2
    return s

Klara will generate test inputs below

import contract_test
def test_foo_0():
    assert contract_test.foo(0, -1.0) == 0
    assert contract_test.foo(0, 0.0) == 0.0

It doesn't generate input v1 > 10000, so the test case would not be able to find out the exceptions. This is because the s at line 3 is unused in the return value.

If we modify the second if statement to elif, which we'll be able to return the [s]{.title-ref} at line 3, klara will generate test inputs that cover v1 > 10000 case.

This is an important distinction with other automatic test case generation available now, because by only generate test case for return values, we can generate a minimal test case, and it's easier to customize how do Klara cover the function.

For example, say we are composing a complex system

    def main(number: int, cm: int, dc: int, wn: int):
        mc = 0
        if wn > 2:
            if number > 2 and number > 2 or number > 2:
                if number > 0:
                    if wn > 2 or wn > 2:
                        mc = 2
                    else:
                        mc = 5
                else:
                    mc = 100
        else:
            mc = 1
        nnn = number * cm
        if cm <= 4:
            num_incr = 4
        else:
            num_incr = cm
        n_num_incr = nnn / num_incr
        nnn_left = dc * num_incr * (n_num_incr / 2 + n_num_incr % 2)
        nnn_right = nnn - nnn_left
        is_flag = nnn_right
        if is_flag:
            cell = Component(nnn_right, options=[mc])
        else:
            cell = Component(nnn_right)
        return cell

It isn't immediately clear to us how many possible return values there are. But we can utilize Klara to generate inputs instantly, below is the generated test

import contract_test
def test_main_0():
    assert contract_test.main(2, 4, 1, 3) is not None
    assert contract_test.main(2, 4, -1, 6) is not None
    assert contract_test.main(2, 4, 1, 4) is not None
    assert contract_test.main(-2, 4, 3, 4) is not None
    assert contract_test.main(-1, -1, -1, 2) is not None
    assert contract_test.main(0, 0, 0, 3) is not None
    assert contract_test.main(0, 0, 0, 6) is not None
    assert contract_test.main(0, 0, 0, 4) is not None
    assert contract_test.main(-2, 0, 0, 4) is not None
    assert contract_test.main(0, 0, 0, 0) is not None

Above generated 10 total results, which is product of nnn_right which have 2 possibilities and mc which have 5 possibilities.

Suppose that 10 tests input is too much, and we have determine that the options argument to Component is redundant to test, we can use Klara's custom plugin to selectively determine which part to ignore in test generation. Go to customize coverage strategy for more information.

After we have setup the plugin, Klara will generate following test

import contract_test
def test_main_0():
    assert contract_test.main(1, 3, 0, 0) == 3.0
    assert contract_test.main(0, 0, 0, 0) == 0.0

Which is only 2 combinations of nnn_right

Because Klara can't dynamically execute the code, it will provide extension to specify how to infer specific ast node or user defined type to make Klara 'smarter'. It's described in extending, extending user type and customize coverage strategy.

Contributing

We use Poetry to manage dependencies. After poetry is installed, run:

$ poetry shell
$ poetry install

To run the test case, do:

$ poetry run pytest test

Acknowledgements

  • The architecture of the inference system is largely inspired by Astroid.
  • Special thanks to Dr. Poh for guiding the early stages of the project.

License

This project is licensed under the terms of the GNU Lesser General Public License.

Comments
  • Multiple errors and confusions in the docs

    Multiple errors and confusions in the docs

    From formal, computer science point of view, docs included in the project contain multiple confusing, or just incorrect, statements.

    Meta-issue trying to pinpoint the issues to help a novice reader, who may get misconceptions after reading these docs.

    opened by pfalcon 6
  • Many Errors

    Many Errors

    1. When I run klara on https://github.com/erezsh/runtype/blob/master/runtype/dataclass.py, I get AttributeError: 'JoinedStr' object has no attribute 'statement' :
      File "c:\python38\lib\site-packages\klara\core\cfg.py", line 354, in rename
        super(ParentScopeBlock, self).rename()
      File "c:\python38\lib\site-packages\klara\core\cfg.py", line 191, in rename
        blk.enumerate()
      File "c:\python38\lib\site-packages\klara\core\cfg.py", line 156, in enumerate
        AttributeEnumerator.enumerate(ast_stmt, False, False)
      File "c:\python38\lib\site-packages\klara\core\ssa.py", line 115, in enumerate
        var.convert_to_ssa()
      File "c:\python38\lib\site-packages\klara\core\node_classes.py", line 399, in convert_to_ssa
        stmt = field.statement()
    AttributeError: 'JoinedStr' object has no attribute 'statement'
    
    1. When I run it on https://github.com/lark-parser/lark/blob/master/lark/utils.py, I get KeyError: 'is not'
    ...
      File "c:\python38\lib\site-packages\klara\core\inference.py", line 1146, in infer_compare
        for result in calc_compare(comp, self.ops, context):
      File "c:\python38\lib\site-packages\klara\core\inference.py", line 1195, in calc_compare
        methods = _comp_op_methods(left, comp, op, context)
      File "c:\python38\lib\site-packages\klara\core\inference.py", line 1161, in _comp_op_methods
        method_name=COMP_OP_DUNDER_METHOD[op],
    KeyError: 'is not'
    
    1. When I run it on https://github.com/lark-parser/lark/blob/master/lark/tree.py I get AttributeError: type object 'Del' has no attribute 'targets'
    ...
        return [self._visit_generic(child) for child in node]
      File "c:\python38\lib\site-packages\klara\core\transform.py", line 36, in _visit_generic
        return self._visit(node)
      File "c:\python38\lib\site-packages\klara\core\transform.py", line 24, in _visit
        returned = self._visit_generic(value)
      File "c:\python38\lib\site-packages\klara\core\transform.py", line 36, in _visit_generic
        return self._visit(node)
      File "c:\python38\lib\site-packages\klara\core\transform.py", line 23, in _visit
        value = getattr(node, name)
    AttributeError: type object 'Del' has no attribute 'targets'
    

    I'm sure I can keep going, but let's stop here for now :)

    opened by erezsh 5
  • AttributeError: 'AsyncFunctionDef' object has no attribute 'statement'

    AttributeError: 'AsyncFunctionDef' object has no attribute 'statement'

    Code:

    async def f():
        pass
    

    Error:

    ❯ klara tmp.py
    loaded extension: {'typeshed_stub.py', 'builtin_inference.py', '__init__.py', 'infer_z3.py', '99_math_z3.py'}
    
    using configuration value: 
    {   'config_file': None,
        'eq_neq': False,
        'input_test_file': 'tmp.py',
        'max_inference_value': None,
        'py_version': 3,
        'stubs': [],
        'type_inference': True,
        'typeshed_select': []}
    
    Traceback (most recent call last):
      File "/home/gram/.local/bin/klara", line 8, in <module>
        sys.exit(main())
      File "/home/gram/.local/lib/python3.9/site-packages/klara/contract/__main__.py", line 50, in main
        output_test = run(input_file.read_text(), input_file.stem)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/contract/__main__.py", line 36, in run
        cfg = MANAGER.build_cfg(tree)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/klara_z3/cov_manager.py", line 64, in build_cfg
        c = cfg.Cfg(as_tree)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 509, in __init__
        self.root, _, _ = self.parse(as_tree)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 629, in parse
        head = self.build(basic_block, head, all_tail_list, func_tail_list)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 535, in build
        tail_list, func_tail = meth(block)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 589, in build_module
        head_returned, tail_list, _ = self.parse(self.as_tree.body)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 627, in parse
        for basic_block in basic_block_parser.get_basic_block():
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 840, in get_basic_block
        basic_block_list = self.visit(node)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 821, in visit
        return visitor(ast_node)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 824, in generic_visit
        self._append_cache(ast_node.statement())
    AttributeError: 'AsyncFunctionDef' object has no attribute 'statement'
    
    
    opened by orsinium 3
  • Improve this by defining the identity table as class variable

    Improve this by defining the identity table as class variable

    https://github.com/usagitoneko97/python-ast/blob/78330cbc3d4601160175f5073a6630a157fce6db/A3.LVN/lvn.py#L30

    Unlike the above, a class variable is instantiated only once when the class is created.

    The following shows how to define it as class variable.

    class Lvn:
        identity_expr = {(None, '*', 2):(None, '+', None), (None, '+', 0):('0', '+', None), .....}
    
        # Accessing the table is a bit tricky though since you need this class' handle to access it.
        # To get the current class, use type(self) as shown below.
        def get_alternate_id(self, id):
            return type(self).identity_expr.get(id)
    
    opened by chaosAD 2
  • Broken links to images, etc. in the docs

    Broken links to images, etc. in the docs

    It seems that https://github.com/usagitoneko97/python-static-code-analysis/commit/1595f838b49f364d456a10673d42914f1d859118 broke various links around. E.g. in https://github.com/usagitoneko97/python-static-code-analysis/blob/master/lvn_optimization/readme.md , links to SVG images are broken . E.g. go to https://github.com/usagitoneko97/python-static-code-analysis/tree/master/lvn_optimization#113-algorithm-in-details, "IvnThird" pseudo-link is rendered instead of an image.

    opened by pfalcon 1
  • Replaced with more aesthetic diagrams and reworded the Dominance:Introduction section

    Replaced with more aesthetic diagrams and reworded the Dominance:Introduction section

    I have replaced your old diagrams with aesthetically appealing diagrams and reworded your text. Please have a look.

    Please use my SVG templates to draw/redraw your diagrams.

    Please don't convert images to PNG files. Now GitHub support SVG directly. See my example.

    Please don't use full URL to your images like https://github.com/usagitoneko97/python-ast/blob/master/A4.Cfg/resources/cfg_ssa_intro.svg.png You should use just resources/cfg_ssa_intro.svg.png instead.

    opened by chaosAD 0
  • Edited readme for Python Implementation version 2

    Edited readme for Python Implementation version 2

    Please have a look at my edited readme.md file. I think you have to rewrite section 1.2.2 Converting Back To SSA. It is unclear and also I think the list given there is incorrect.

    opened by chaosAD 0
  • Api documentation

    Api documentation

    Hello, it seems that you api documentation page is empty. Maybe a misconfiguration of the autodoc Sphinx extension.

    As a temporary solution, the docs are available here: https://pydocbrowser.github.io/klara/latest/index.html

    opened by tristanlatr 0
  • Support for imports?

    Support for imports?

    Hello,

    First good job at writing this library. It looks like it has powerful capabilities.

    I was wondering if this library still maintained? Do you planned support for inferring imports anytime soon ?

    I’ve myself implemented a astroid-alike (intra procedural) inference engine: https://github.com/tristanlatr/astuce and it supports imports, except wildcard imports. But klara seems much more intelligent.

    Tell me what you think. Thanks

    opened by tristanlatr 0
  • BREAKING CHANGE: remove py2 support

    BREAKING CHANGE: remove py2 support

    I've removed typed-ast package at there are problems with this package in apple-silicon. Also, typed ast recommends using the builtin ast module from python 3.8 and up.

    I could not understand why the uts are failing, I would like to get help to fix those.

    opened by jochman 3
Releases(0.6.3)
  • 0.6.3(Sep 19, 2021)

    v0.6.3 Release Notes (09/19/21)

    • Fixed conditions in loop causing conflicting conditions propagation (#7)
    • fixed Del and Delete shared the same node, and caused fields error (#7)
    • implemented identity (is, is not) comparison for const and instance (#7)
    • added AsyncFunctionDef, Await, AsyncFor, AsyncWith ast support
    • implemented repr, ascii builtin call, and JoinedStr, FormattedValue inference (#7, #8)

    Fixed #7, #8

    Source code(tar.gz)
    Source code(zip)
Owner
Ho Guo Xian
I like automation.
Ho Guo Xian
API mocking with Python.

apyr apyr (all lowercase) is a simple & easy to use mock API server. It's great for front-end development when your API is not ready, or when you are

Umut Seven 55 Nov 25, 2022
Rerun pytest when your code changes

A simple watcher for pytest Overview pytest-watcher is a tool to automatically rerun pytest when your code changes. It looks for the following events:

Olzhas Arystanov 74 Dec 29, 2022
A simple python script that uses selenium(chrome web driver),pyautogui,time and schedule modules to enter google meets automatically

A simple python script that uses selenium(chrome web driver),pyautogui,time and schedule modules to enter google meets automatically

3 Feb 07, 2022
Green is a clean, colorful, fast python test runner.

Green -- A clean, colorful, fast python test runner. Features Clean - Low redundancy in output. Result statistics for each test is vertically aligned.

Nathan Stocks 756 Dec 22, 2022
This repository contains a set of benchmarks of different implementations of Parquet (storage format) <-> Arrow (in-memory format).

Parquet benchmarks This repository contains a set of benchmarks of different implementations of Parquet (storage format) - Arrow (in-memory format).

11 Dec 21, 2022
A rewrite of Python's builtin doctest module (with pytest plugin integration) but without all the weirdness

The xdoctest package is a re-write of Python's builtin doctest module. It replaces the old regex-based parser with a new abstract-syntax-tree based pa

Jon Crall 174 Dec 16, 2022
Python Rest Testing

pyresttest Table of Contents What Is It? Status Installation Sample Test Examples Installation How Do I Use It? Running A Simple Test Using JSON Valid

Sam Van Oort 1.1k Dec 28, 2022
Integration layer between Requests and Selenium for automation of web actions.

Requestium is a Python library that merges the power of Requests, Selenium, and Parsel into a single integrated tool for automatizing web actions. The

Tryolabs 1.7k Dec 27, 2022
Silky smooth profiling for Django

Silk Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before prese

Jazzband 3.7k Jan 04, 2023
buX Course Enrollment Automation

buX automation BRACU - buX course enrollment automation Features: Automatically enroll into multiple courses at a time. Find courses just entering cou

Mohammad Shakib 1 Oct 06, 2022
Object factory for Django

Model Bakery: Smart fixtures for better tests Model Bakery offers you a smart way to create fixtures for testing in Django. With a simple and powerful

Model Bakers 632 Jan 08, 2023
Implement unittest, removing all global variable and returning values

Implement unittest, removing all global variable and returning values

Placide 1 Nov 01, 2021
WIP SAT benchmarking tooling, written with only my personal use in mind.

SAT Benchmarking Some early work in progress tooling for running benchmarks and keeping track of the results when working on SAT solvers and related t

Jannis Harder 1 Dec 26, 2021
Redis fixtures and fixture factories for Pytest.

Redis fixtures and fixture factories for Pytest.This is a pytest plugin, that enables you to test your code that relies on a running Redis database. It allows you to specify additional fixtures for R

Clearcode 86 Dec 23, 2022
This package is a python library with tools for the Molecular Simulation - Software Gromos.

This package is a python library with tools for the Molecular Simulation - Software Gromos. It allows you to easily set up, manage and analyze simulations in python.

14 Sep 28, 2022
py.test fixture for benchmarking code

Overview docs tests package A pytest fixture for benchmarking code. It will group the tests into rounds that are calibrated to the chosen timer. See c

Ionel Cristian Mărieș 1k Jan 03, 2023
A simple Python script I wrote that scrapes NASA's James Webb Space Telescope tracker website using Selenium and returns its current status and location.

A simple Python script I wrote that scrapes NASA's James Webb Space Telescope tracker website using Selenium and returns its current status and location.

9 Feb 10, 2022
Language-agnostic HTTP API Testing Tool

Dredd — HTTP API Testing Framework Dredd is a language-agnostic command-line tool for validating API description document against backend implementati

Apiary 4k Jan 05, 2023
Nokia SR OS automation

Nokia SR OS automation Nokia is one of the biggest vendors of the telecommunication equipment, which is very popular in the Service Provider segment.

Karneliuk.com 7 Jul 23, 2022
UX Analytics & A/B Testing

UX Analytics & A/B Testing

Marvin EDORH 1 Sep 07, 2021