The Python agent for Apache SkyWalking

Overview

SkyWalking Python Agent

Sky Walking logo

SkyWalking-Python: The Python Agent for Apache SkyWalking, which provides the native tracing abilities for Python project.

SkyWalking: an APM(application performance monitor) system, especially designed for microservices, cloud native and container-based (Docker, Kubernetes, Mesos) architectures.

GitHub stars Twitter Follow

Build

Documentation

Installation Requirements

SkyWalking Python Agent requires SkyWalking 8.0+ and Python 3.6+.

If you would like to try out the latest features that are not released yet, please refer to this guide to build from sources.

Contributing

Before submitting a pull request or pushing a commit, please read our contributing and developer guide.

Contact Us

License

Apache 2.0

Comments
  • feat: Add support for MeterReportService

    feat: Add support for MeterReportService

    Add support for MeterReportService

    Behavior

    This service runs along with the agent like other services and reports roughly every 20 seconds.

    Syntactic sugar

    Counter

    c = Counter('c2', CounterMode.INCREMENT)
    c.build()
    
    # increase Counter c by the time the with-wrapped codes consumed
    with c.create_timer():
        # some codes may consume a certain time
    
    c = Counter('c3', CounterMode.INCREMENT)
    c.build()
    
    # increase Counter c by num once counter_decorator_test gets called
    @Counter.increase(name='c3', num=2)
    def counter_decorator_test():
        # some codes
    
    c = Counter('c4', CounterMode.INCREMENT)
    c.build()
    
    # increase Counter c by the time counter_decorator_test consumed
    @Counter.timer(name='c4')
    def counter_decorator_test(s):
        # some codes may consume a certain time
    

    Histogram

    h = Histogram('h3', [i / 10 for i in range(10)])
    h.build()
    
    # Histogram h will record the time the with-wrapped codes consumed
    with h.create_timer():
        # some codes may consume a certain time
    
    h = Histogram('h2', [i / 10 for i in range(10)])
    h.build()
    
    # Histogram h will record the time histogram_decorator_test consumed
    @Histogram.timer(name='h2')
    def histogram_decorator_test(s):
        time.sleep(s)
    

    Simple test

    insert following codes below this line agent/__init__.py#L151

    from skywalking.meter.gauge import Gauge
    import gc
    
    def generator_for_gc_g0():
        while(True):
            yield gc.get_stats()[0]['collected']
    
    def generator_for_gc_g1():
        while(True):
            yield gc.get_stats()[1]['collected']
    
    def generator_for_gc_g2():
        while(True):
            yield gc.get_stats()[2]['collected']
    
    
    gc_g0_meter = Gauge("gc_g0", generator_for_gc_g0())
    gc_g0_meter.build()
    
    gc_g1_meter = Gauge("gc_g1", generator_for_gc_g1())
    gc_g1_meter.build()
    
    gc_g2_meter = Gauge("gc_g2", generator_for_gc_g2())
    gc_g2_meter.build()
    
    feature core 
    opened by jiang1997 19
  • fix: grpc timeout segment data loss

    fix: grpc timeout segment data loss

    Please review this thoroughly because all I wanted to do was resend segments which failed to send due to a grpc timeout. But unexpectedly the behavior of grpc timeout changed, it doesn't time out anymore, the heartbeat seems to keep it alive now. I don't understand why this behavior changed so that is why I am requesting a thorough looking at this.

    enhancement 
    opened by tom-pytel 18
  • Add http ignore by method

    Add http ignore by method

    • Added SW_HTTP_IGNORE_METHOD which will allow ignoring http operations by method (GET, POST, HEAD, OPTIONS, etc...)
    • Set default SW_AGENT_MAX_BUFFER_SIZE back to 10000 from 1000, which turns out to be too small for python (many unnecessarily dropped spans).
    feature 
    opened by tom-pytel 14
  • feat: report metrics related to pvm

    feat: report metrics related to pvm

    Metrics

    CPU

    total_cpu_utilization

    process_cpu_utilization

    cpu utilization of the current process

    thread_active_count

    number of threads invoked by the current process and its children

    Memory

    total_mem_utilization

    process_mem_utilization

    memory utilization of the current process

    GC

    gc_g0

    number of objects of generation 0

    gc_g1

    number of objects of generation 1

    gc_g2

    number of objects of generation 2

    gc_time

    For now, MeterReportService is available with gRPC only. So does the e2e test. I will implement MeterReportService and enable e2e test for Kafka later.

    Welcome any advice about naming.

    feature core 
    opened by jiang1997 12
  • [Fix][Plugin] sw_flask general exceptions handled

    [Fix][Plugin] sw_flask general exceptions handled

    • sw_flask fix will handle errors like returning the wrong type from a handler or other internal errors.
    • Updated StackedSpan to track depth, and made depth variable instance instead of class level (this was a bug).
    • Changed how SpanContext decides when all spans finished to write Segment data, now counts span start / stops which should work better across different async scenarios.
    • Changed new_exit_span() with span.inject() to work simpler like the NodeJS agent, now plugins inject directly themselves if they need to.
    • Removed carrier from plugins which didn't actually use it.
    bug plugin 
    opened by tom-pytel 12
  • Add plugin for bottle

    Add plugin for bottle

    • [x] Add a test case for the new plugin
    • [x] Add a CHANGELOG entry for the new plugin
    • [x] Add a component id in the main repo
    • [x] Add a logo in the UI repo
    • [x] Rebuild the requirements.txt by running tools/env/build_requirements_(linux|windows).sh
    • [x] Rebuild the Plugins.md documentation by running make doc-gen
    plugin 
    opened by jiang1997 11
  • Enable HTTP log reporting

    Enable HTTP log reporting

    This is a work in progress.

    Now reporting logs in JSON through HTTP to http://oap/v3/logs does work.

    But, I'm not sure if the oap/v3/logs endpoint is for such usage(It seems designed for fluent-bit batch reporting?). Reporting logs one by one through HTTP may not be ideal in terms of performance. I'm not sure whether the Java agent only implements gRPC reporter intentionally out of this reason.

    Please advise.

    Signed-off-by: YihaoChen [email protected]

    feature 
    opened by Superskyyy 11
  • Feature: collect and report logs

    Feature: collect and report logs

    This is a OSPP Summer 2021 project supervised by @Humbertzhang | apache/skywalking#7118

    The feature implements optional log reporter functionalities in alignment with the SkyWalking Java agent.

    • Intercepts logs from Python logging module.
    • Reports logs via a new temporary gRPC channel(to be removed in the future).
    • Supports unformatted/ formatted logs with custom layout.
    • Supports custom logging level threshold.
    • Bumps up submodule to support log collection protocols.
    • Bumps up skywalking-eyes and adds a config entry to ignore .gitignore during license checks.
    feature core 
    opened by Superskyyy 11
  • Introduce better coding style

    Introduce better coding style

    This PR migrates existing usage of old-style string manipulation (%, .format and +) to f-strings for better clarity and some performance boost.

    Adds stricter coding style.

    Optimizes debug messages for performance.

    Signed-off-by: Superskyyy [email protected]

    enhancement chore 
    opened by Superskyyy 10
  • Add E2E coverage for Trace and Logging

    Add E2E coverage for Trace and Logging

    • Change Fastapi to FastAPI component name (sync with OAP change)
    • Add E2E coverage for Python agent (Trace and Logging), profiling is not tested for now.
    • Renamed a build variable to remove possible mix-ups with CLI environment Var.
    • Enhance plugin test to reuse images.
    • Combined CI to pass if changes are non-essential to agent and workflows.

    Note: There's a flaky test due to a possible {{- contains }} issue in the Infra-e2e verifier, the logs seem to be queried in an unstable order by the CLI where later log often placed first in log list, combining with the contains bug, it becomes flaky.

    A time.sleep() workaround makes sure the log arrives in clear order which passes the e2e for now.

    test 
    opened by Superskyyy 9
  • [Enhancement] Async tasks should work 100%

    [Enhancement] Async tasks should work 100%

    These changes should address all remaining async-related problems not handled in #88. The spans list has been moved from an instance variable in SpanContext to a global task-local variable which allows "forking" it for new subtasks so that multiple concurrent async subtasks don't interfere with one another. spans is now a top-level module var because Python stipulates that ContextVar should be such and only one context exists at any given time so it is essentially a singleton anyway.

    enhancement core 
    opened by tom-pytel 9
  • Enhance redis plugin to adapt Virtual Cache

    Enhance redis plugin to adapt Virtual Cache

    Resolves: #10212

    enhancement 
    opened by Jedore 2
Releases(v0.8.0)
  • v0.8.0(Jul 10, 2022)

    What's Changed

    • spans correctly reference finished parents by @tom-pytel in https://github.com/apache/skywalking-python/pull/161
    • Refactor current Python agent docs to serve on SkyWalking official website by @Superskyyy in https://github.com/apache/skywalking-python/pull/162
    • fix broken url by @JaredTan95 in https://github.com/apache/skywalking-python/pull/163
    • Remove docs from main README.md - Add website doc links. by @Superskyyy in https://github.com/apache/skywalking-python/pull/164
    • Refactor SkyWalking Python to use the CLI for CI instead of legacy setup by @Superskyyy in https://github.com/apache/skywalking-python/pull/165
    • Remove places that mentions Python 3.5 support due to EOL by @Superskyyy in https://github.com/apache/skywalking-python/pull/166
    • Cleanup and Python 3.10 support by @Superskyyy in https://github.com/apache/skywalking-python/pull/167
    • filled out rest of psycopg 3 plugin by @tom-pytel in https://github.com/apache/skywalking-python/pull/168
    • Move flake configs all together by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/169
    • Enable faster CI by categorical parallelism by @Superskyyy in https://github.com/apache/skywalking-python/pull/170
    • Introduce better coding style by @Superskyyy in https://github.com/apache/skywalking-python/pull/171
    • Minimize exclude in lint by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/173
    • Introduce another set of flake8 extensions by @Superskyyy in https://github.com/apache/skywalking-python/pull/174
    • fix aiohttp outgoing request url by @tom-pytel in https://github.com/apache/skywalking-python/pull/175
    • bugfix: flask + nginx got KeyError: 'REMOTE_PORT' in sw_flask.py plugin by @VxCoder in https://github.com/apache/skywalking-python/pull/176
    • Add missing ending quote by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/177
    • updated deprecated configuration by @doddi in https://github.com/apache/skywalking-python/pull/179
    • Add plugin for mysqlclient by @katelei6 in https://github.com/apache/skywalking-python/pull/178
    • Fix sw-rabbitmq TypeError when there are no headers by @dcryans in https://github.com/apache/skywalking-python/pull/182
    • Fix agent bootup traceback not shown in sw-python CLI by @Superskyyy in https://github.com/apache/skywalking-python/pull/183
    • Add plugin for FastAPI by @katelei6 in https://github.com/apache/skywalking-python/pull/181
    • Support CI container logs by @Superskyyy in https://github.com/apache/skywalking-python/pull/185
    • Update mySQL plugin to support two different parameter keys. by @katelei6 in https://github.com/apache/skywalking-python/pull/186
    • add the code > 400 is error by @katelei6 in https://github.com/apache/skywalking-python/pull/187
    • Doc: add how to use with uwsgi by @arcosx in https://github.com/apache/skywalking-python/pull/188
    • Fix: right doc link for How To Use With uWSGI by @arcosx in https://github.com/apache/skywalking-python/pull/189
    • The local log stack depth is not truncated by @wzy960520 in https://github.com/apache/skywalking-python/pull/190
    • Revert "The local log stack depth is not truncated (#190)" by @Superskyyy in https://github.com/apache/skywalking-python/pull/191
    • Fix sw_formatter wrongly set cache that impacts user handlers by @Superskyyy in https://github.com/apache/skywalking-python/pull/192
    • Fix typo that cause failure in loading user sitecustomize.py by @Superskyyy in https://github.com/apache/skywalking-python/pull/193
    • Drop support for Flask 1.x due to EOL & Jinja2 issue by @Superskyyy in https://github.com/apache/skywalking-python/pull/195
    • Update agent docs, changelogs and PR template. by @Superskyyy in https://github.com/apache/skywalking-python/pull/197
    • Fix Python shown as UNKNOWN language in OAP by @Superskyyy in https://github.com/apache/skywalking-python/pull/194
    • Fix logging level not properly set according to config by @Superskyyy in https://github.com/apache/skywalking-python/pull/196
    • Fix the properties are not set correctly by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/198
    • Add E2E coverage for Trace and Logging by @Superskyyy in https://github.com/apache/skywalking-python/pull/199
    • Support log reporter safe mode by @Superskyyy in https://github.com/apache/skywalking-python/pull/200
    • Fix scheduled event fails changes CI by @Superskyyy in https://github.com/apache/skywalking-python/pull/201
    • Fix deadlink and CI on schedule by @Superskyyy in https://github.com/apache/skywalking-python/pull/203
    • Remove namespace to instance properties and add pid by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/205
    • Enhance Traceback depth default to 10 by @Superskyyy in https://github.com/apache/skywalking-python/pull/206
    • Unify the tag name with other agents by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/208
    • Improved ignore path regex by @tom-pytel in https://github.com/apache/skywalking-python/pull/210
    • Fix sw_psycopg2 register_type() by @tom-pytel in https://github.com/apache/skywalking-python/pull/211
    • Fix psycopg2 register_type() second arg default by @tom-pytel in https://github.com/apache/skywalking-python/pull/212
    • Add plugin doc check in CI by @JarvisG495 in https://github.com/apache/skywalking-python/pull/213
    • Fix typo in docker/Makefile by @jiang1997 in https://github.com/apache/skywalking-python/pull/216
    • Update UI repository link in PR template by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/217
    • Add plugin for bottle by @jiang1997 in https://github.com/apache/skywalking-python/pull/214
    • Draft release 0.8.0 by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/219
    • Migrate license checker to license-eye and adjust the release script by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/220

    New Contributors

    • @JaredTan95 made their first contribution in https://github.com/apache/skywalking-python/pull/163
    • @VxCoder made their first contribution in https://github.com/apache/skywalking-python/pull/176
    • @doddi made their first contribution in https://github.com/apache/skywalking-python/pull/179
    • @katelei6 made their first contribution in https://github.com/apache/skywalking-python/pull/178
    • @dcryans made their first contribution in https://github.com/apache/skywalking-python/pull/182
    • @arcosx made their first contribution in https://github.com/apache/skywalking-python/pull/188
    • @wzy960520 made their first contribution in https://github.com/apache/skywalking-python/pull/190
    • @JarvisG495 made their first contribution in https://github.com/apache/skywalking-python/pull/213
    • @jiang1997 made their first contribution in https://github.com/apache/skywalking-python/pull/216

    Full Changelog: https://github.com/apache/skywalking-python/compare/v0.7.0...v0.8.0

    Source code(tar.gz)
    Source code(zip)
  • v0.7.0(Sep 13, 2021)

    • Feature:

      • Support collecting and reporting logs to backend (#147)
      • Support profiling Python method level performance (#127
      • Add a new sw-python CLI that enables agent non-intrusive integration (#156)
      • Add exponential reconnection backoff strategy when OAP is down (#157)
      • Support ignoring traces by http method (#143)
      • NoopSpan on queue full, propagation downstream (#141)
      • Support agent namespace. (#126)
      • Support secure connection option for GRPC and HTTP (#134)
    • Plugins:

      • Add Falcon Plugin (#146)
      • Update sw_pymongo.py to be compatible with cluster mode (#150)
      • Add Python celery plugin (#125)
      • Support tornado5+ and tornado6+ (#119)
    • Fixes:

      • Remove HTTP basic auth credentials from log, stacktrace, segment (#152)
      • Fix @trace decorator not work (#136)
      • Fix grpc disconnect, add SW_AGENT_MAX_BUFFER_SIZE to control buffer queue size (#138)
    • Others:

      • Chore: bump up requests version to avoid license issue (#142)
      • Fix module wrapt as normal install dependency (#123)
      • Explicit component inheritance (#132)
      • Provide dockerfile & images for easy integration in containerized scenarios (#159)
    Source code(tar.gz)
    Source code(zip)
  • v0.6.0(Mar 31, 2021)

    • Fixes:
      • Segment data loss when gRPC timing out. (#116)
      • sw_tornado plugin async handler status set correctly. (#115)
      • sw_pymysql error when connection haven't db. (#113)
    Source code(tar.gz)
    Source code(zip)
  • v0.5.0(Dec 31, 2020)

    • New plugins

      • Pyramid Plugin (#102)
      • AioHttp Plugin (#101)
      • Sanic Plugin (#91)
    • API and enhancements

      • @trace decorator supports async functions
      • Supports async task context
      • Optimized path trace ignore
      • Moved exception check to Span.__exit__
      • Moved Method & Url tags before requests
    • Fixes:

      • BaseExceptions not recorded as errors
      • Allow pending data to send before exit
      • sw_flask general exceptions handled
      • Make skywalking logging Non-global
    • Chores and tests

      • Make tests really run on specified Python version
      • Deprecate 3.5 as it's EOL
    Source code(tar.gz)
    Source code(zip)
  • v0.4.0(Nov 24, 2020)

    • Feature: Support Kafka reporter protocol (#74)
    • BugFix: Move generated packages into skywalking namespace to avoid conflicts (#72)
    • BugFix: Agent cannot reconnect after server is down (#79)
    • Test: Mitigate unsafe yaml loading (#76)
    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Aug 28, 2020)

    • New plugins

      • Urllib3 Plugin (#69)
      • Elasticsearch Plugin (#64)
      • PyMongo Plugin (#60)
      • Rabbitmq Plugin (#53)
      • Make plugin compatible with Django (#52)
    • API

      • Add process propagation (#67)
      • Add tags to decorators (#65)
      • Add Check version of packages when install plugins (#63)
      • Add thread propagation (#62)
      • Add trace ignore (#59)
      • Support snapshot context (#56)
      • Support correlation context (#55)
    • Chores and tests

      • Test: run multiple versions of supported libraries (#66)
      • Chore: add pull request template for plugin (#61)
      • Chore: add dev doc and reorganize the structure (#58)
      • Test: update test health check (#57)
      • Chore: add make goal to package release tar ball (#54)
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Jul 28, 2020)

    • New plugins

      • Kafka Plugin (#50)
      • Tornado Plugin (#48)
      • Redis Plugin (#44)
      • Django Plugin (#37)
      • PyMsql Plugin (#35)
      • Flask plugin (#31)
    • API

      • Add ignore_suffix Config (#40)
      • Add missing log method and simplify test codes (#34)
      • Add content equality of SegmentRef (#30)
      • Validate carrier before using it (#29)
    • Chores and tests

      • Test: print the diff list when validation failed (#46)
      • Created venv builders for linux/windows and req flashers + use documentation (#38)
    Source code(tar.gz)
    Source code(zip)
  • v0.1.0(Jun 28, 2020)

Owner
The Apache Software Foundation
The Apache Software Foundation
GWCelery is a simple and reliable package for annotating and orchestrating LIGO/Virgo alerts

GWCelery is a simple and reliable package for annotating and orchestrating LIGO/Virgo alerts, built from widely used open source components.

Min-A Cho Zeno 1 Nov 02, 2021
Create Arrays (Working with For Loops)

DSA with Python Create Arrays (Working with For Loops) CREATING ARRAYS WITH USER INPUT Array is a collection of items stored at contiguous memory loca

1 Feb 08, 2022
Collaboration project to creating bank application maded by Anzhelica Sakun and Yuriy Konyukh

Collaboration project to creating bank application maded by Anzhelica Sakun and Yuriy Konyukh

Yuriy 1 Jan 08, 2022
A simply dashboard to view commodities position data based on CFTC reports

commodities-dashboard A simply dashboard to view commodities position data based on CFTC reports This is a python project using Dash and plotly to con

71 Dec 19, 2022
Shell Trality API for local development.

Trality Simulator Intro This package is a work in progress. It allows local development of Trality bots in an IDE such as VS Code. The package provide

CrypTrality 1 Nov 17, 2021
Twikoo自定义表情列表 | HexoPlusPlus自定义表情列表(其实基于OwO的项目都可以用的啦)

Twikoo-Magic 更新说明 2021/1/15 基于2021/1/14 Twikoo 更新1.1.0-beta,所有表情都将以缩写形式(如:[ text ]:)输出。1/14之前本仓库有部分表情text缺失及重复, 导致无法正常使用表情 1/14后的所有表情json列表已全部更新

noionion 90 Jan 05, 2023
Dicionario-git-github - Dictionary created to help train new users of Git and GitHub applications

Dicionário 📕 Dicionário criado com o objetivo de auxiliar no treinamento de nov

Felippe Rafael 1 Feb 07, 2022
A site that went kinda viral that lets you put Bernie Sanders in places

Bernie In Places An app that accidentally went viral! Read the story in WIRED here Install First, create a python virtual environment, and install all

310 Aug 22, 2022
Here, I have discuss the three methods of list reversion. The three methods are built-in method, slicing method and position changing method.

Three-different-method-for-list-reversion Here, I have discuss the three methods of list reversion. The three methods are built-in method, slicing met

Sachin Vinayak Dabhade 4 Sep 24, 2021
Interfaces between napari and pymeshlab library to allow import, export and construction of surfaces.

napari-pymeshlab Interfaces between napari and the pymeshlab library to allow import, export and construction of surfaces. This is a WIP and feature r

Zach Marin 4 Oct 12, 2022
MODSKIN-LOLPRO-updater: The mod is fkn 10y old and has'nt a self-updater

The mod is fkn 10y old and has'nt a self-updater. To use it just run the exec, wait some seconds, and it will run the new modsk

Shiro Amurha 3 Apr 23, 2022
Python samples for Google Cloud Platform products.

Google Cloud Platform Python Samples Python samples for Google Cloud Platform products. Setup Install pip and virtualenv if you do not already have th

Google Cloud Platform 6k Jan 03, 2023
Practice in Oxford_AI&ML class

Practice in Oxford_AI&ML class

St3ve Lee 2 Feb 04, 2022
Async Python Circuit Breaker implementation

aiocircuitbreaker This is an async Python implementation of the circuitbreaker library. Installation The project is available on PyPI. Simply run: $ p

5 Sep 05, 2022
Arcpy Tool developed for ArcMap 10.x that checks DVOF points against TDS data and creates an output feature class as well as a check database.

DVOF_check_tool Arcpy Tool developed for ArcMap 10.x that checks DVOF points against TDS data and creates an output feature class as well as a check d

3 Apr 18, 2022
Dapp / Forge traces enhancer

traces-explorer Dapp / Forge traces enhancer Usage traces.py and pattern_* files should be in the same directory make test traces.txt py traces.

1 Feb 02, 2022
北大选课网2021年春季验证码识别

北大选课网验证码识别 2021 年春季学期 Powered by Elector Quartet (@Rabbit, @xmcp, @SpiritedAwayCN, @gzz) 数据集描述 最初的数据集为 5130 张人工标记的验证码,之后利用早期训练好的模型在选课网上进行自动验证 (自举),又收集

Rabbit 27 Sep 17, 2022
A promo calculator for sports betting odds.

Sportbetter Calculation Toolkit Parlay Calculator This is a quick parlay calculator that considers some of the common promos offered. It is used to id

Luke Bhan 1 Sep 08, 2022
SkyPort console user terminal written in python

SkyPort terminal implemented as a console script written in Python Description Sky Port is an universal bus between user software and compute resource

Sky Workflows 1 Oct 23, 2022
Functional collections extension functions for Python

pyfuncol pyfuncol Installation Usage API Documentation Compatibility Contributing License A Python functional collections library. It extends collecti

Andrea Veneziano 32 Nov 16, 2022