The Python agent for Apache SkyWalking

Overview

SkyWalking Python Agent

Sky Walking logo

SkyWalking-Python: The Python Agent for Apache SkyWalking, which provides the native tracing abilities for Python project.

SkyWalking: an APM(application performance monitor) system, especially designed for microservices, cloud native and container-based (Docker, Kubernetes, Mesos) architectures.

GitHub stars Twitter Follow

Build

Documentation

Installation Requirements

SkyWalking Python Agent requires SkyWalking 8.0+ and Python 3.6+.

If you would like to try out the latest features that are not released yet, please refer to this guide to build from sources.

Contributing

Before submitting a pull request or pushing a commit, please read our contributing and developer guide.

Contact Us

License

Apache 2.0

Comments
  • feat: Add support for MeterReportService

    feat: Add support for MeterReportService

    Add support for MeterReportService

    Behavior

    This service runs along with the agent like other services and reports roughly every 20 seconds.

    Syntactic sugar

    Counter

    c = Counter('c2', CounterMode.INCREMENT)
    c.build()
    
    # increase Counter c by the time the with-wrapped codes consumed
    with c.create_timer():
        # some codes may consume a certain time
    
    c = Counter('c3', CounterMode.INCREMENT)
    c.build()
    
    # increase Counter c by num once counter_decorator_test gets called
    @Counter.increase(name='c3', num=2)
    def counter_decorator_test():
        # some codes
    
    c = Counter('c4', CounterMode.INCREMENT)
    c.build()
    
    # increase Counter c by the time counter_decorator_test consumed
    @Counter.timer(name='c4')
    def counter_decorator_test(s):
        # some codes may consume a certain time
    

    Histogram

    h = Histogram('h3', [i / 10 for i in range(10)])
    h.build()
    
    # Histogram h will record the time the with-wrapped codes consumed
    with h.create_timer():
        # some codes may consume a certain time
    
    h = Histogram('h2', [i / 10 for i in range(10)])
    h.build()
    
    # Histogram h will record the time histogram_decorator_test consumed
    @Histogram.timer(name='h2')
    def histogram_decorator_test(s):
        time.sleep(s)
    

    Simple test

    insert following codes below this line agent/__init__.py#L151

    from skywalking.meter.gauge import Gauge
    import gc
    
    def generator_for_gc_g0():
        while(True):
            yield gc.get_stats()[0]['collected']
    
    def generator_for_gc_g1():
        while(True):
            yield gc.get_stats()[1]['collected']
    
    def generator_for_gc_g2():
        while(True):
            yield gc.get_stats()[2]['collected']
    
    
    gc_g0_meter = Gauge("gc_g0", generator_for_gc_g0())
    gc_g0_meter.build()
    
    gc_g1_meter = Gauge("gc_g1", generator_for_gc_g1())
    gc_g1_meter.build()
    
    gc_g2_meter = Gauge("gc_g2", generator_for_gc_g2())
    gc_g2_meter.build()
    
    feature core 
    opened by jiang1997 19
  • fix: grpc timeout segment data loss

    fix: grpc timeout segment data loss

    Please review this thoroughly because all I wanted to do was resend segments which failed to send due to a grpc timeout. But unexpectedly the behavior of grpc timeout changed, it doesn't time out anymore, the heartbeat seems to keep it alive now. I don't understand why this behavior changed so that is why I am requesting a thorough looking at this.

    enhancement 
    opened by tom-pytel 18
  • Add http ignore by method

    Add http ignore by method

    • Added SW_HTTP_IGNORE_METHOD which will allow ignoring http operations by method (GET, POST, HEAD, OPTIONS, etc...)
    • Set default SW_AGENT_MAX_BUFFER_SIZE back to 10000 from 1000, which turns out to be too small for python (many unnecessarily dropped spans).
    feature 
    opened by tom-pytel 14
  • feat: report metrics related to pvm

    feat: report metrics related to pvm

    Metrics

    CPU

    total_cpu_utilization

    process_cpu_utilization

    cpu utilization of the current process

    thread_active_count

    number of threads invoked by the current process and its children

    Memory

    total_mem_utilization

    process_mem_utilization

    memory utilization of the current process

    GC

    gc_g0

    number of objects of generation 0

    gc_g1

    number of objects of generation 1

    gc_g2

    number of objects of generation 2

    gc_time

    For now, MeterReportService is available with gRPC only. So does the e2e test. I will implement MeterReportService and enable e2e test for Kafka later.

    Welcome any advice about naming.

    feature core 
    opened by jiang1997 12
  • [Fix][Plugin] sw_flask general exceptions handled

    [Fix][Plugin] sw_flask general exceptions handled

    • sw_flask fix will handle errors like returning the wrong type from a handler or other internal errors.
    • Updated StackedSpan to track depth, and made depth variable instance instead of class level (this was a bug).
    • Changed how SpanContext decides when all spans finished to write Segment data, now counts span start / stops which should work better across different async scenarios.
    • Changed new_exit_span() with span.inject() to work simpler like the NodeJS agent, now plugins inject directly themselves if they need to.
    • Removed carrier from plugins which didn't actually use it.
    bug plugin 
    opened by tom-pytel 12
  • Add plugin for bottle

    Add plugin for bottle

    • [x] Add a test case for the new plugin
    • [x] Add a CHANGELOG entry for the new plugin
    • [x] Add a component id in the main repo
    • [x] Add a logo in the UI repo
    • [x] Rebuild the requirements.txt by running tools/env/build_requirements_(linux|windows).sh
    • [x] Rebuild the Plugins.md documentation by running make doc-gen
    plugin 
    opened by jiang1997 11
  • Enable HTTP log reporting

    Enable HTTP log reporting

    This is a work in progress.

    Now reporting logs in JSON through HTTP to http://oap/v3/logs does work.

    But, I'm not sure if the oap/v3/logs endpoint is for such usage(It seems designed for fluent-bit batch reporting?). Reporting logs one by one through HTTP may not be ideal in terms of performance. I'm not sure whether the Java agent only implements gRPC reporter intentionally out of this reason.

    Please advise.

    Signed-off-by: YihaoChen [email protected]

    feature 
    opened by Superskyyy 11
  • Feature: collect and report logs

    Feature: collect and report logs

    This is a OSPP Summer 2021 project supervised by @Humbertzhang | apache/skywalking#7118

    The feature implements optional log reporter functionalities in alignment with the SkyWalking Java agent.

    • Intercepts logs from Python logging module.
    • Reports logs via a new temporary gRPC channel(to be removed in the future).
    • Supports unformatted/ formatted logs with custom layout.
    • Supports custom logging level threshold.
    • Bumps up submodule to support log collection protocols.
    • Bumps up skywalking-eyes and adds a config entry to ignore .gitignore during license checks.
    feature core 
    opened by Superskyyy 11
  • Introduce better coding style

    Introduce better coding style

    This PR migrates existing usage of old-style string manipulation (%, .format and +) to f-strings for better clarity and some performance boost.

    Adds stricter coding style.

    Optimizes debug messages for performance.

    Signed-off-by: Superskyyy [email protected]

    enhancement chore 
    opened by Superskyyy 10
  • Add E2E coverage for Trace and Logging

    Add E2E coverage for Trace and Logging

    • Change Fastapi to FastAPI component name (sync with OAP change)
    • Add E2E coverage for Python agent (Trace and Logging), profiling is not tested for now.
    • Renamed a build variable to remove possible mix-ups with CLI environment Var.
    • Enhance plugin test to reuse images.
    • Combined CI to pass if changes are non-essential to agent and workflows.

    Note: There's a flaky test due to a possible {{- contains }} issue in the Infra-e2e verifier, the logs seem to be queried in an unstable order by the CLI where later log often placed first in log list, combining with the contains bug, it becomes flaky.

    A time.sleep() workaround makes sure the log arrives in clear order which passes the e2e for now.

    test 
    opened by Superskyyy 9
  • [Enhancement] Async tasks should work 100%

    [Enhancement] Async tasks should work 100%

    These changes should address all remaining async-related problems not handled in #88. The spans list has been moved from an instance variable in SpanContext to a global task-local variable which allows "forking" it for new subtasks so that multiple concurrent async subtasks don't interfere with one another. spans is now a top-level module var because Python stipulates that ContextVar should be such and only one context exists at any given time so it is essentially a singleton anyway.

    enhancement core 
    opened by tom-pytel 9
  • Enhance redis plugin to adapt Virtual Cache

    Enhance redis plugin to adapt Virtual Cache

    Resolves: #10212

    enhancement 
    opened by Jedore 2
Releases(v0.8.0)
  • v0.8.0(Jul 10, 2022)

    What's Changed

    • spans correctly reference finished parents by @tom-pytel in https://github.com/apache/skywalking-python/pull/161
    • Refactor current Python agent docs to serve on SkyWalking official website by @Superskyyy in https://github.com/apache/skywalking-python/pull/162
    • fix broken url by @JaredTan95 in https://github.com/apache/skywalking-python/pull/163
    • Remove docs from main README.md - Add website doc links. by @Superskyyy in https://github.com/apache/skywalking-python/pull/164
    • Refactor SkyWalking Python to use the CLI for CI instead of legacy setup by @Superskyyy in https://github.com/apache/skywalking-python/pull/165
    • Remove places that mentions Python 3.5 support due to EOL by @Superskyyy in https://github.com/apache/skywalking-python/pull/166
    • Cleanup and Python 3.10 support by @Superskyyy in https://github.com/apache/skywalking-python/pull/167
    • filled out rest of psycopg 3 plugin by @tom-pytel in https://github.com/apache/skywalking-python/pull/168
    • Move flake configs all together by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/169
    • Enable faster CI by categorical parallelism by @Superskyyy in https://github.com/apache/skywalking-python/pull/170
    • Introduce better coding style by @Superskyyy in https://github.com/apache/skywalking-python/pull/171
    • Minimize exclude in lint by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/173
    • Introduce another set of flake8 extensions by @Superskyyy in https://github.com/apache/skywalking-python/pull/174
    • fix aiohttp outgoing request url by @tom-pytel in https://github.com/apache/skywalking-python/pull/175
    • bugfix: flask + nginx got KeyError: 'REMOTE_PORT' in sw_flask.py plugin by @VxCoder in https://github.com/apache/skywalking-python/pull/176
    • Add missing ending quote by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/177
    • updated deprecated configuration by @doddi in https://github.com/apache/skywalking-python/pull/179
    • Add plugin for mysqlclient by @katelei6 in https://github.com/apache/skywalking-python/pull/178
    • Fix sw-rabbitmq TypeError when there are no headers by @dcryans in https://github.com/apache/skywalking-python/pull/182
    • Fix agent bootup traceback not shown in sw-python CLI by @Superskyyy in https://github.com/apache/skywalking-python/pull/183
    • Add plugin for FastAPI by @katelei6 in https://github.com/apache/skywalking-python/pull/181
    • Support CI container logs by @Superskyyy in https://github.com/apache/skywalking-python/pull/185
    • Update mySQL plugin to support two different parameter keys. by @katelei6 in https://github.com/apache/skywalking-python/pull/186
    • add the code > 400 is error by @katelei6 in https://github.com/apache/skywalking-python/pull/187
    • Doc: add how to use with uwsgi by @arcosx in https://github.com/apache/skywalking-python/pull/188
    • Fix: right doc link for How To Use With uWSGI by @arcosx in https://github.com/apache/skywalking-python/pull/189
    • The local log stack depth is not truncated by @wzy960520 in https://github.com/apache/skywalking-python/pull/190
    • Revert "The local log stack depth is not truncated (#190)" by @Superskyyy in https://github.com/apache/skywalking-python/pull/191
    • Fix sw_formatter wrongly set cache that impacts user handlers by @Superskyyy in https://github.com/apache/skywalking-python/pull/192
    • Fix typo that cause failure in loading user sitecustomize.py by @Superskyyy in https://github.com/apache/skywalking-python/pull/193
    • Drop support for Flask 1.x due to EOL & Jinja2 issue by @Superskyyy in https://github.com/apache/skywalking-python/pull/195
    • Update agent docs, changelogs and PR template. by @Superskyyy in https://github.com/apache/skywalking-python/pull/197
    • Fix Python shown as UNKNOWN language in OAP by @Superskyyy in https://github.com/apache/skywalking-python/pull/194
    • Fix logging level not properly set according to config by @Superskyyy in https://github.com/apache/skywalking-python/pull/196
    • Fix the properties are not set correctly by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/198
    • Add E2E coverage for Trace and Logging by @Superskyyy in https://github.com/apache/skywalking-python/pull/199
    • Support log reporter safe mode by @Superskyyy in https://github.com/apache/skywalking-python/pull/200
    • Fix scheduled event fails changes CI by @Superskyyy in https://github.com/apache/skywalking-python/pull/201
    • Fix deadlink and CI on schedule by @Superskyyy in https://github.com/apache/skywalking-python/pull/203
    • Remove namespace to instance properties and add pid by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/205
    • Enhance Traceback depth default to 10 by @Superskyyy in https://github.com/apache/skywalking-python/pull/206
    • Unify the tag name with other agents by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/208
    • Improved ignore path regex by @tom-pytel in https://github.com/apache/skywalking-python/pull/210
    • Fix sw_psycopg2 register_type() by @tom-pytel in https://github.com/apache/skywalking-python/pull/211
    • Fix psycopg2 register_type() second arg default by @tom-pytel in https://github.com/apache/skywalking-python/pull/212
    • Add plugin doc check in CI by @JarvisG495 in https://github.com/apache/skywalking-python/pull/213
    • Fix typo in docker/Makefile by @jiang1997 in https://github.com/apache/skywalking-python/pull/216
    • Update UI repository link in PR template by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/217
    • Add plugin for bottle by @jiang1997 in https://github.com/apache/skywalking-python/pull/214
    • Draft release 0.8.0 by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/219
    • Migrate license checker to license-eye and adjust the release script by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/220

    New Contributors

    • @JaredTan95 made their first contribution in https://github.com/apache/skywalking-python/pull/163
    • @VxCoder made their first contribution in https://github.com/apache/skywalking-python/pull/176
    • @doddi made their first contribution in https://github.com/apache/skywalking-python/pull/179
    • @katelei6 made their first contribution in https://github.com/apache/skywalking-python/pull/178
    • @dcryans made their first contribution in https://github.com/apache/skywalking-python/pull/182
    • @arcosx made their first contribution in https://github.com/apache/skywalking-python/pull/188
    • @wzy960520 made their first contribution in https://github.com/apache/skywalking-python/pull/190
    • @JarvisG495 made their first contribution in https://github.com/apache/skywalking-python/pull/213
    • @jiang1997 made their first contribution in https://github.com/apache/skywalking-python/pull/216

    Full Changelog: https://github.com/apache/skywalking-python/compare/v0.7.0...v0.8.0

    Source code(tar.gz)
    Source code(zip)
  • v0.7.0(Sep 13, 2021)

    • Feature:

      • Support collecting and reporting logs to backend (#147)
      • Support profiling Python method level performance (#127
      • Add a new sw-python CLI that enables agent non-intrusive integration (#156)
      • Add exponential reconnection backoff strategy when OAP is down (#157)
      • Support ignoring traces by http method (#143)
      • NoopSpan on queue full, propagation downstream (#141)
      • Support agent namespace. (#126)
      • Support secure connection option for GRPC and HTTP (#134)
    • Plugins:

      • Add Falcon Plugin (#146)
      • Update sw_pymongo.py to be compatible with cluster mode (#150)
      • Add Python celery plugin (#125)
      • Support tornado5+ and tornado6+ (#119)
    • Fixes:

      • Remove HTTP basic auth credentials from log, stacktrace, segment (#152)
      • Fix @trace decorator not work (#136)
      • Fix grpc disconnect, add SW_AGENT_MAX_BUFFER_SIZE to control buffer queue size (#138)
    • Others:

      • Chore: bump up requests version to avoid license issue (#142)
      • Fix module wrapt as normal install dependency (#123)
      • Explicit component inheritance (#132)
      • Provide dockerfile & images for easy integration in containerized scenarios (#159)
    Source code(tar.gz)
    Source code(zip)
  • v0.6.0(Mar 31, 2021)

    • Fixes:
      • Segment data loss when gRPC timing out. (#116)
      • sw_tornado plugin async handler status set correctly. (#115)
      • sw_pymysql error when connection haven't db. (#113)
    Source code(tar.gz)
    Source code(zip)
  • v0.5.0(Dec 31, 2020)

    • New plugins

      • Pyramid Plugin (#102)
      • AioHttp Plugin (#101)
      • Sanic Plugin (#91)
    • API and enhancements

      • @trace decorator supports async functions
      • Supports async task context
      • Optimized path trace ignore
      • Moved exception check to Span.__exit__
      • Moved Method & Url tags before requests
    • Fixes:

      • BaseExceptions not recorded as errors
      • Allow pending data to send before exit
      • sw_flask general exceptions handled
      • Make skywalking logging Non-global
    • Chores and tests

      • Make tests really run on specified Python version
      • Deprecate 3.5 as it's EOL
    Source code(tar.gz)
    Source code(zip)
  • v0.4.0(Nov 24, 2020)

    • Feature: Support Kafka reporter protocol (#74)
    • BugFix: Move generated packages into skywalking namespace to avoid conflicts (#72)
    • BugFix: Agent cannot reconnect after server is down (#79)
    • Test: Mitigate unsafe yaml loading (#76)
    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Aug 28, 2020)

    • New plugins

      • Urllib3 Plugin (#69)
      • Elasticsearch Plugin (#64)
      • PyMongo Plugin (#60)
      • Rabbitmq Plugin (#53)
      • Make plugin compatible with Django (#52)
    • API

      • Add process propagation (#67)
      • Add tags to decorators (#65)
      • Add Check version of packages when install plugins (#63)
      • Add thread propagation (#62)
      • Add trace ignore (#59)
      • Support snapshot context (#56)
      • Support correlation context (#55)
    • Chores and tests

      • Test: run multiple versions of supported libraries (#66)
      • Chore: add pull request template for plugin (#61)
      • Chore: add dev doc and reorganize the structure (#58)
      • Test: update test health check (#57)
      • Chore: add make goal to package release tar ball (#54)
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Jul 28, 2020)

    • New plugins

      • Kafka Plugin (#50)
      • Tornado Plugin (#48)
      • Redis Plugin (#44)
      • Django Plugin (#37)
      • PyMsql Plugin (#35)
      • Flask plugin (#31)
    • API

      • Add ignore_suffix Config (#40)
      • Add missing log method and simplify test codes (#34)
      • Add content equality of SegmentRef (#30)
      • Validate carrier before using it (#29)
    • Chores and tests

      • Test: print the diff list when validation failed (#46)
      • Created venv builders for linux/windows and req flashers + use documentation (#38)
    Source code(tar.gz)
    Source code(zip)
  • v0.1.0(Jun 28, 2020)

Owner
The Apache Software Foundation
The Apache Software Foundation
Code and yara rules to detect and analyze Cobalt Strike

Cobalt Strike Resources This repository contains: analyze.py: a script to analyze a Cobalt Strike beacon (python analyze.py BEACON) extract.py; extrac

Tek 224 Jan 04, 2023
Type Persian without confusing words for yourself and others, in Adobe Connect

About In the Adobe Connect chat section, to type in Persian or Arabic, the written words will be confused and will be written and sent illegibly (This

Matin Najafi 23 Nov 26, 2021
Scitizen - Help scientific research for the benefit of mankind and humanity πŸ”¬

Scitizen - Help scientific research for the benefit of mankind and humanity πŸ”¬ Scitizen has been built from the ground up to give everyone the possibi

Pierre CORBEL 21 Mar 08, 2022
A variant caller for the GBA gene using WGS data

Gauchian: WGS-based GBA variant caller Gauchian is a targeted variant caller for the GBA gene based on a whole-genome sequencing (WGS) BAM file. Gauch

Illumina 16 Oct 13, 2022
БСрвис слуТит прокси ΠΌΠ΅ΠΆΠ΄Ρƒ cСрвисом рСгистрации ошибок ΠΏΠ»Π°Ρ‚Ρ„ΠΎΡ€ΠΌΡ‹ ΠΈ систСмой сбора ошибок Sentry

Sentry Reg Service БСрвис слуТит прокси ΠΌΠ΅ΠΆΠ΄Ρƒ CСрвисом рСгистрации ошибок ΠΏΠ»Π°Ρ‚Ρ„ΠΎΡ€ΠΌΡ‹ ΠΈ систСмой сбора ошибок Sentry. Как Ρ€Π°Π·Π²Π΅Ρ€Π½ΡƒΡ‚ΡŒ Sentry onpremise. Π‘

Ingvar Vilkman 13 May 24, 2022
You will need to install a few python packages for this one.

Features Bait support Auto repair will repair every 10 catches Anti detection (still a work in progress) but using random times and click positions Pr

12 Sep 21, 2022
A free website that keeps the people informed about housing and evictions.

Eviction Tracker Currently helping verify detainer warrant data for middle Tennessee - via Middle TN DSA - Red Door Collective Features Presents data

Red Door Collective 7 Dec 14, 2022
Org agenda in the console

This Python script reads an org agenda file (i.e. a regular org file with some active dates) and displays an interactive and colored year calendar with detailed information for each day when the mous

Nicolas P. Rougier 113 Jan 03, 2023
python3 scrip for case conversion of source code files writen in fixed form fortran

convert_FORTRAN_case python3 scrip for case conversion of source code files writen in fixed form fortran python3 scrip for case conversion of source c

7 Sep 20, 2022
πŸ”΅Open many google dorks in a fasted way

Dorkinho πŸ”΅ The author is not responsible for misuse of the tool, use it in good practices like Pentest and CTF OSINT challenges. Dorkinho is a script

SidHawks 2 May 02, 2022
Hello, Welcome to this repo. don't forget to read guidelines in readme.md

Hacktoberfest_2021 If you looking for your first contribution, we are here to help. Just create a simple program using any language you like in our fo

Wafa Rifqi Anafin 117 Dec 14, 2022
Intelligent Employer Profiling Platform.

Intelligent Employer Profiling Platform Setup Instructions Generating Model Data Ensure that Python 3.9+ and pip is installed. Install project depende

Harvey Donnelly 2 Jan 09, 2022
Ultimate Microsoft Edge Uninstaller!

Ultimate Microsoft Edge Uninstaller

1 Feb 08, 2022
This simple script generates a backup of a given Python and R environment

Python Environment Backup It’s always good to maintain your Python and R Anaconda environment packages properly listed and well-kept in case you have

Andrew Laganaro 1 Jul 13, 2022
Voldemort's Python import helper

importmagician Voldemort's Python import helper pip install importmagician Import from uninstalled Python directories Say you have a directory (relat

Zhengyang Feng 4 Mar 09, 2022
Earth-to-orbit ballistic trajectories with atmospheric resistance

Earth-to-orbit ballistic trajectories with atmospheric resistance Overview Space guns are a theoretical technology that reduces the cost of getting bu

1 Dec 03, 2021
Repository, with small useful and functional applications

Repositorio,com pequenos aplicativos uteis e funcionais A ideia e ir deselvolvendo pequenos aplicativos funcionais e adicionar a este repositorio List

GabrielDuke 6 Dec 06, 2021
Este script aΓ±ade la config de s4vitar a bspwm automaticamente!

Se ha testeado este script en ParrotOS, Kali y Ubuntu. Funciona para todos los sistemas operativos basados en Debian. InstalaciΓ³n git clone https://gi

yorkox 201 Dec 30, 2022
Small exercises to get you used to reading and writing Python code!

Pythonlings Welcome to Pythonlings, an automated Python tutorial program (inspired by Rustlings and Haskellings). WIP This program is still working in

ιΉ€ηΏ”δΈ‡ι‡Œ 5 Sep 23, 2022
Completed task 1 and task 2 at LetsGrowMore as a data science intern.

LetsGrowMore-Internship Completed task 1 and task 2 at LetsGrowMore as a data science intern. Task 1- Task 2- Creating a Decision Tree classifier and

Sanjyot Panure 1 Jan 16, 2022