Klara is a static analysis tools to automatic generate test case, based on SMT (z3) solver, with a powerful ast level inference system.

Overview

Klara

Klara is a static analysis tools to automatic generate test case, based on SMT (z3) solver, with a powerful ast level inference system. Klara will take python file as input and generate corresponding test file in pytest format, that attempt to cover all return values. For example, following function in file test.py

def triangle(x: int, y: int, z: int) -> str:
    if x == y == z:
        return "Equilateral triangle"
    elif x == y or y == z or x == z:
        return "Isosceles triangle"
    else:
        return "Scalene triangle"

will generate

import test
def test_triangle_0():
    assert test.triangle(0, 0, 0) == 'Equilateral triangle'
    assert test.triangle(0, 0, 1) == 'Isosceles triangle'
    assert test.triangle(2, 0, 1) == 'Scalene triangle'

See the Klara's documentation at https://klara-py.readthedocs.io

Installing

Klara can be installed via pip tool by using:

pip install klara

Usage

We can invoke klara on any python source file, and it will generate a corresponding pytest test file.

$ cat source.py
def foo(x: int, y: int, z: str):
    if x + y > 2:
        return x + y + 12
    elif x < y:
        return x + y
    elif (z + "me") == "some":
        return z + "thing"
    else:
        return x - y

$ klara source.py
$ cat test_source.py
import contract_test


def test_foo_0():
    assert contract_test.foo(0, 3, \'\') == 15
    assert contract_test.foo(0, 1, \'\') == 1
    assert contract_test.foo(0, 0, \'so\') == \'sothing\'
    assert contract_test.foo(0, 0, \'\') == 0

Consult the quick start manual for more examples and guidance. To use it as a static analysis library, go to Inference.

Why Klara?

Klara works on ast level and it doesn't execute user code in any way, which is a very important difference compared to similar tool like Crosshair and Pynguin that utilize concolic symbolic execution that required user code execution that might cause unwanted side effects. Klara work on ast level, combine with data flow analysis that utilize Control Flow Graph(CFG), Static Single Assignment(SSA), use-def chain, etc... to build a powerful python inference system that leverages Z3-solver for constraints solving and path feasibility check. Because of this, Klara is able to operate on both python2/3 source code with the help of typed_ast. To specify the source code is in python 2, pass in -py 2 argument. It's python 3 by default.

Klara can also be used as a static analysis tool, allow user to define custom rule to identify programming bugs, error or enforcing coding standard. With SMT solver support, analysis will be more accurate and greatly reduce false-positive case. For example

4: if v1 < 3: z = 1 else: z = 2 else: z = 3 s = z """) with klara.MANAGER.initialize_z3_var_from_func(tree.body[0]): print(list(tree.body[0].body[-1].value.infer())) ">
import klara
tree = klara.parse("""
    def foo(v1: int):
        if v1 > 4:
            if v1 < 3:
                z = 1
            else:
                z = 2
        else:
            z = 3
        s = z
""")
with klara.MANAGER.initialize_z3_var_from_func(tree.body[0]):
    print(list(tree.body[0].body[-1].value.infer()))

Will print out:

[2, 3]

Because z = 1 is not possible due to v1 > 4 and v1 < 3 is unsatisfiable

The inference system architecture and api is largely inspired by Astroid, a static inference library used by Pylint.

Klara utilize the inference system to generate test case, in other words, it generate test case for all possible return values of the function, instead of generate test case for all control path of the function.

To illustrate the point, consider the function below, with divide by zero vulnerabilities at line 3

def foo(v1: int, v2: float):
    if v1 > 10000:
        s = v1 / 0  # unused statement
    if v1 > v2:
        s = v1
    else:
        s = v2
    return s

Klara will generate test inputs below

import contract_test
def test_foo_0():
    assert contract_test.foo(0, -1.0) == 0
    assert contract_test.foo(0, 0.0) == 0.0

It doesn't generate input v1 > 10000, so the test case would not be able to find out the exceptions. This is because the s at line 3 is unused in the return value.

If we modify the second if statement to elif, which we'll be able to return the [s]{.title-ref} at line 3, klara will generate test inputs that cover v1 > 10000 case.

This is an important distinction with other automatic test case generation available now, because by only generate test case for return values, we can generate a minimal test case, and it's easier to customize how do Klara cover the function.

For example, say we are composing a complex system

    def main(number: int, cm: int, dc: int, wn: int):
        mc = 0
        if wn > 2:
            if number > 2 and number > 2 or number > 2:
                if number > 0:
                    if wn > 2 or wn > 2:
                        mc = 2
                    else:
                        mc = 5
                else:
                    mc = 100
        else:
            mc = 1
        nnn = number * cm
        if cm <= 4:
            num_incr = 4
        else:
            num_incr = cm
        n_num_incr = nnn / num_incr
        nnn_left = dc * num_incr * (n_num_incr / 2 + n_num_incr % 2)
        nnn_right = nnn - nnn_left
        is_flag = nnn_right
        if is_flag:
            cell = Component(nnn_right, options=[mc])
        else:
            cell = Component(nnn_right)
        return cell

It isn't immediately clear to us how many possible return values there are. But we can utilize Klara to generate inputs instantly, below is the generated test

import contract_test
def test_main_0():
    assert contract_test.main(2, 4, 1, 3) is not None
    assert contract_test.main(2, 4, -1, 6) is not None
    assert contract_test.main(2, 4, 1, 4) is not None
    assert contract_test.main(-2, 4, 3, 4) is not None
    assert contract_test.main(-1, -1, -1, 2) is not None
    assert contract_test.main(0, 0, 0, 3) is not None
    assert contract_test.main(0, 0, 0, 6) is not None
    assert contract_test.main(0, 0, 0, 4) is not None
    assert contract_test.main(-2, 0, 0, 4) is not None
    assert contract_test.main(0, 0, 0, 0) is not None

Above generated 10 total results, which is product of nnn_right which have 2 possibilities and mc which have 5 possibilities.

Suppose that 10 tests input is too much, and we have determine that the options argument to Component is redundant to test, we can use Klara's custom plugin to selectively determine which part to ignore in test generation. Go to customize coverage strategy for more information.

After we have setup the plugin, Klara will generate following test

import contract_test
def test_main_0():
    assert contract_test.main(1, 3, 0, 0) == 3.0
    assert contract_test.main(0, 0, 0, 0) == 0.0

Which is only 2 combinations of nnn_right

Because Klara can't dynamically execute the code, it will provide extension to specify how to infer specific ast node or user defined type to make Klara 'smarter'. It's described in extending, extending user type and customize coverage strategy.

Contributing

We use Poetry to manage dependencies. After poetry is installed, run:

$ poetry shell
$ poetry install

To run the test case, do:

$ poetry run pytest test

Acknowledgements

  • The architecture of the inference system is largely inspired by Astroid.
  • Special thanks to Dr. Poh for guiding the early stages of the project.

License

This project is licensed under the terms of the GNU Lesser General Public License.

Comments
  • Multiple errors and confusions in the docs

    Multiple errors and confusions in the docs

    From formal, computer science point of view, docs included in the project contain multiple confusing, or just incorrect, statements.

    Meta-issue trying to pinpoint the issues to help a novice reader, who may get misconceptions after reading these docs.

    opened by pfalcon 6
  • Many Errors

    Many Errors

    1. When I run klara on https://github.com/erezsh/runtype/blob/master/runtype/dataclass.py, I get AttributeError: 'JoinedStr' object has no attribute 'statement' :
      File "c:\python38\lib\site-packages\klara\core\cfg.py", line 354, in rename
        super(ParentScopeBlock, self).rename()
      File "c:\python38\lib\site-packages\klara\core\cfg.py", line 191, in rename
        blk.enumerate()
      File "c:\python38\lib\site-packages\klara\core\cfg.py", line 156, in enumerate
        AttributeEnumerator.enumerate(ast_stmt, False, False)
      File "c:\python38\lib\site-packages\klara\core\ssa.py", line 115, in enumerate
        var.convert_to_ssa()
      File "c:\python38\lib\site-packages\klara\core\node_classes.py", line 399, in convert_to_ssa
        stmt = field.statement()
    AttributeError: 'JoinedStr' object has no attribute 'statement'
    
    1. When I run it on https://github.com/lark-parser/lark/blob/master/lark/utils.py, I get KeyError: 'is not'
    ...
      File "c:\python38\lib\site-packages\klara\core\inference.py", line 1146, in infer_compare
        for result in calc_compare(comp, self.ops, context):
      File "c:\python38\lib\site-packages\klara\core\inference.py", line 1195, in calc_compare
        methods = _comp_op_methods(left, comp, op, context)
      File "c:\python38\lib\site-packages\klara\core\inference.py", line 1161, in _comp_op_methods
        method_name=COMP_OP_DUNDER_METHOD[op],
    KeyError: 'is not'
    
    1. When I run it on https://github.com/lark-parser/lark/blob/master/lark/tree.py I get AttributeError: type object 'Del' has no attribute 'targets'
    ...
        return [self._visit_generic(child) for child in node]
      File "c:\python38\lib\site-packages\klara\core\transform.py", line 36, in _visit_generic
        return self._visit(node)
      File "c:\python38\lib\site-packages\klara\core\transform.py", line 24, in _visit
        returned = self._visit_generic(value)
      File "c:\python38\lib\site-packages\klara\core\transform.py", line 36, in _visit_generic
        return self._visit(node)
      File "c:\python38\lib\site-packages\klara\core\transform.py", line 23, in _visit
        value = getattr(node, name)
    AttributeError: type object 'Del' has no attribute 'targets'
    

    I'm sure I can keep going, but let's stop here for now :)

    opened by erezsh 5
  • AttributeError: 'AsyncFunctionDef' object has no attribute 'statement'

    AttributeError: 'AsyncFunctionDef' object has no attribute 'statement'

    Code:

    async def f():
        pass
    

    Error:

    ❯ klara tmp.py
    loaded extension: {'typeshed_stub.py', 'builtin_inference.py', '__init__.py', 'infer_z3.py', '99_math_z3.py'}
    
    using configuration value: 
    {   'config_file': None,
        'eq_neq': False,
        'input_test_file': 'tmp.py',
        'max_inference_value': None,
        'py_version': 3,
        'stubs': [],
        'type_inference': True,
        'typeshed_select': []}
    
    Traceback (most recent call last):
      File "/home/gram/.local/bin/klara", line 8, in <module>
        sys.exit(main())
      File "/home/gram/.local/lib/python3.9/site-packages/klara/contract/__main__.py", line 50, in main
        output_test = run(input_file.read_text(), input_file.stem)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/contract/__main__.py", line 36, in run
        cfg = MANAGER.build_cfg(tree)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/klara_z3/cov_manager.py", line 64, in build_cfg
        c = cfg.Cfg(as_tree)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 509, in __init__
        self.root, _, _ = self.parse(as_tree)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 629, in parse
        head = self.build(basic_block, head, all_tail_list, func_tail_list)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 535, in build
        tail_list, func_tail = meth(block)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 589, in build_module
        head_returned, tail_list, _ = self.parse(self.as_tree.body)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 627, in parse
        for basic_block in basic_block_parser.get_basic_block():
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 840, in get_basic_block
        basic_block_list = self.visit(node)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 821, in visit
        return visitor(ast_node)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 824, in generic_visit
        self._append_cache(ast_node.statement())
    AttributeError: 'AsyncFunctionDef' object has no attribute 'statement'
    
    
    opened by orsinium 3
  • Improve this by defining the identity table as class variable

    Improve this by defining the identity table as class variable

    https://github.com/usagitoneko97/python-ast/blob/78330cbc3d4601160175f5073a6630a157fce6db/A3.LVN/lvn.py#L30

    Unlike the above, a class variable is instantiated only once when the class is created.

    The following shows how to define it as class variable.

    class Lvn:
        identity_expr = {(None, '*', 2):(None, '+', None), (None, '+', 0):('0', '+', None), .....}
    
        # Accessing the table is a bit tricky though since you need this class' handle to access it.
        # To get the current class, use type(self) as shown below.
        def get_alternate_id(self, id):
            return type(self).identity_expr.get(id)
    
    opened by chaosAD 2
  • Broken links to images, etc. in the docs

    Broken links to images, etc. in the docs

    It seems that https://github.com/usagitoneko97/python-static-code-analysis/commit/1595f838b49f364d456a10673d42914f1d859118 broke various links around. E.g. in https://github.com/usagitoneko97/python-static-code-analysis/blob/master/lvn_optimization/readme.md , links to SVG images are broken . E.g. go to https://github.com/usagitoneko97/python-static-code-analysis/tree/master/lvn_optimization#113-algorithm-in-details, "IvnThird" pseudo-link is rendered instead of an image.

    opened by pfalcon 1
  • Replaced with more aesthetic diagrams and reworded the Dominance:Introduction section

    Replaced with more aesthetic diagrams and reworded the Dominance:Introduction section

    I have replaced your old diagrams with aesthetically appealing diagrams and reworded your text. Please have a look.

    Please use my SVG templates to draw/redraw your diagrams.

    Please don't convert images to PNG files. Now GitHub support SVG directly. See my example.

    Please don't use full URL to your images like https://github.com/usagitoneko97/python-ast/blob/master/A4.Cfg/resources/cfg_ssa_intro.svg.png You should use just resources/cfg_ssa_intro.svg.png instead.

    opened by chaosAD 0
  • Edited readme for Python Implementation version 2

    Edited readme for Python Implementation version 2

    Please have a look at my edited readme.md file. I think you have to rewrite section 1.2.2 Converting Back To SSA. It is unclear and also I think the list given there is incorrect.

    opened by chaosAD 0
  • Api documentation

    Api documentation

    Hello, it seems that you api documentation page is empty. Maybe a misconfiguration of the autodoc Sphinx extension.

    As a temporary solution, the docs are available here: https://pydocbrowser.github.io/klara/latest/index.html

    opened by tristanlatr 0
  • Support for imports?

    Support for imports?

    Hello,

    First good job at writing this library. It looks like it has powerful capabilities.

    I was wondering if this library still maintained? Do you planned support for inferring imports anytime soon ?

    I’ve myself implemented a astroid-alike (intra procedural) inference engine: https://github.com/tristanlatr/astuce and it supports imports, except wildcard imports. But klara seems much more intelligent.

    Tell me what you think. Thanks

    opened by tristanlatr 0
  • BREAKING CHANGE: remove py2 support

    BREAKING CHANGE: remove py2 support

    I've removed typed-ast package at there are problems with this package in apple-silicon. Also, typed ast recommends using the builtin ast module from python 3.8 and up.

    I could not understand why the uts are failing, I would like to get help to fix those.

    opened by jochman 3
Releases(0.6.3)
  • 0.6.3(Sep 19, 2021)

    v0.6.3 Release Notes (09/19/21)

    • Fixed conditions in loop causing conflicting conditions propagation (#7)
    • fixed Del and Delete shared the same node, and caused fields error (#7)
    • implemented identity (is, is not) comparison for const and instance (#7)
    • added AsyncFunctionDef, Await, AsyncFor, AsyncWith ast support
    • implemented repr, ascii builtin call, and JoinedStr, FormattedValue inference (#7, #8)

    Fixed #7, #8

    Source code(tar.gz)
    Source code(zip)
Owner
Ho Guo Xian
I like automation.
Ho Guo Xian
This package is a python library with tools for the Molecular Simulation - Software Gromos.

This package is a python library with tools for the Molecular Simulation - Software Gromos. It allows you to easily set up, manage and analyze simulations in python.

14 Sep 28, 2022
Test for generating stylized circuit traces from images

I test of an image processing idea to take an image and make neat circuit board art automatically. Inspired by this twitter post by @JackRhysider

Miller Hooks 3 Dec 12, 2022
Aioresponses is a helper for mock/fake web requests in python aiohttp package.

aioresponses Aioresponses is a helper to mock/fake web requests in python aiohttp package. For requests module there are a lot of packages that help u

402 Jan 06, 2023
Argument matchers for unittest.mock

callee Argument matchers for unittest.mock More robust tests Python's mocking library (or its backport for Python 3.3) is simple, reliable, and easy

Karol Kuczmarski 77 Nov 03, 2022
API mocking with Python.

apyr apyr (all lowercase) is a simple & easy to use mock API server. It's great for front-end development when your API is not ready, or when you are

Umut Seven 55 Nov 25, 2022
The evaluator covering all of the metrics required by tasks within the DUE Benchmark.

DUE Evaluator The repository contains the evaluator covering all of the metrics required by tasks within the DUE Benchmark, i.e., set-based F1 (for KI

DUE Benchmark 4 Jan 21, 2022
Implement unittest, removing all global variable and returning values

Implement unittest, removing all global variable and returning values

Placide 1 Nov 01, 2021
Bayesian A/B testing

bayesian_testing is a small package for a quick evaluation of A/B (or A/B/C/...) tests using Bayesian approach.

Matus Baniar 35 Dec 15, 2022
A folder automation made using Watch-dog, it only works in linux for now but I assume, it will be adaptable to mac and PC as well

folder-automation A folder automation made using Watch-dog, it only works in linux for now but I assume, it will be adaptable to mac and PC as well Th

Parag Jyoti Paul 31 May 28, 2021
Tools for test driven data-wrangling and data validation.

datatest: Test driven data-wrangling and data validation Datatest helps to speed up and formalize data-wrangling and data validation tasks. It impleme

269 Dec 16, 2022
A command-line tool and Python library and Pytest plugin for automated testing of RESTful APIs, with a simple, concise and flexible YAML-based syntax

1.0 Release See here for details about breaking changes with the upcoming 1.0 release: https://github.com/taverntesting/tavern/issues/495 Easier API t

909 Dec 15, 2022
Local continuous test runner with pytest and watchdog.

pytest-watch -- Continuous pytest runner pytest-watch a zero-config CLI tool that runs pytest, and re-runs it when a file in your project changes. It

Joe Esposito 675 Dec 23, 2022
:game_die: Pytest plugin to randomly order tests and control random.seed

pytest-randomly Pytest plugin to randomly order tests and control random.seed. Features All of these features are on by default but can be disabled wi

pytest-dev 471 Dec 30, 2022
Testing - Instrumenting Sanic framework with Opentelemetry

sanic-otel-splunk Testing - Instrumenting Sanic framework with Opentelemetry Test with python 3.8.10, sanic 20.12.2 Step to instrument pip install -r

Donler 1 Nov 26, 2021
Docker-based integration tests

Docker-based integration tests Description Simple pytest fixtures that help you write integration tests with Docker and docker-compose. Specify all ne

Avast 326 Dec 27, 2022
A pure Python script to easily get a reverse shell

easy-shell A pure Python script to easily get a reverse shell. How it works? After sending a request, it generates a payload with different commands a

Cristian Souza 48 Dec 12, 2022
Compiles python selenium script to be a Window's executable

Problem Statement Setting up a Python project can be frustrating for non-developers. From downloading the right version of python, setting up virtual

Jerry Ng 8 Jan 09, 2023
Plugin for generating HTML reports for pytest results

pytest-html pytest-html is a plugin for pytest that generates a HTML report for test results. Resources Documentation Release Notes Issue Tracker Code

pytest-dev 548 Dec 28, 2022
Yet another python home automation project. Because a smart light is more than just on or off

Automate home Yet another home automation project because a smart light is more than just on or off. Overview When talking about home automation there

Maja Massarini 62 Oct 10, 2022
Getting the most out of your hobby servo

ServoProject by Adam Bäckström Getting the most out of your hobby servo Theory The control system of a regular hobby servo looks something like this:

209 Dec 20, 2022