The Python agent for Apache SkyWalking

Last update: Dec 12, 2022

Overview

SkyWalking Python Agent

SkyWalking-Python: The Python Agent for Apache SkyWalking, which provides the native tracing abilities for Python project.

SkyWalking: an APM(application performance monitor) system, especially designed for microservices, cloud native and container-based (Docker, Kubernetes, Mesos) architectures.

Documentation

Official documentation
Blog about the Python Agent Profiling Feature

Installation Requirements

SkyWalking Python Agent requires SkyWalking 8.0+ and Python 3.6+.

If you would like to try out the latest features that are not released yet, please refer to this guide to build from sources.

Contributing

Before submitting a pull request or pushing a commit, please read our contributing and developer guide.

Contact Us

Submit an GitHub Issue by using [Python] as title prefix.
Mail list: [email protected]. Mail to [email protected], follow the reply to subscribe the mail list.
Join skywalking channel at Apache Slack. If the link is not working, find the latest one at Apache INFRA WIKI.
Twitter, ASFSkyWalking

License

Apache 2.0

Comments

feat: Add support for MeterReportService

Add support for MeterReportService

Behavior

This service runs along with the agent like other services and reports roughly every 20 seconds.

Syntactic sugar

Counter

c = Counter('c2', CounterMode.INCREMENT)
c.build()

# increase Counter c by the time the with-wrapped codes consumed
with c.create_timer():
    # some codes may consume a certain time

c = Counter('c3', CounterMode.INCREMENT)
c.build()

# increase Counter c by num once counter_decorator_test gets called
@Counter.increase(name='c3', num=2)
def counter_decorator_test():
    # some codes

c = Counter('c4', CounterMode.INCREMENT)
c.build()

# increase Counter c by the time counter_decorator_test consumed
@Counter.timer(name='c4')
def counter_decorator_test(s):
    # some codes may consume a certain time

Histogram

h = Histogram('h3', [i / 10 for i in range(10)])
h.build()

# Histogram h will record the time the with-wrapped codes consumed
with h.create_timer():
    # some codes may consume a certain time

h = Histogram('h2', [i / 10 for i in range(10)])
h.build()

# Histogram h will record the time histogram_decorator_test consumed
@Histogram.timer(name='h2')
def histogram_decorator_test(s):
    time.sleep(s)

Simple test

insert following codes below this line agent/__init__.py#L151

from skywalking.meter.gauge import Gauge
import gc

def generator_for_gc_g0():
    while(True):
        yield gc.get_stats()[0]['collected']

def generator_for_gc_g1():
    while(True):
        yield gc.get_stats()[1]['collected']

def generator_for_gc_g2():
    while(True):
        yield gc.get_stats()[2]['collected']


gc_g0_meter = Gauge("gc_g0", generator_for_gc_g0())
gc_g0_meter.build()

gc_g1_meter = Gauge("gc_g1", generator_for_gc_g1())
gc_g1_meter.build()

gc_g2_meter = Gauge("gc_g2", generator_for_gc_g2())
gc_g2_meter.build()

feature core

opened by jiang1997 19

fix: grpc timeout segment data loss

Please review this thoroughly because all I wanted to do was resend segments which failed to send due to a grpc timeout. But unexpectedly the behavior of grpc timeout changed, it doesn't time out anymore, the heartbeat seems to keep it alive now. I don't understand why this behavior changed so that is why I am requesting a thorough looking at this.
enhancement

opened by tom-pytel 18
Add http ignore by method
Added SW_HTTP_IGNORE_METHOD which will allow ignoring http operations by method (GET, POST, HEAD, OPTIONS, etc...)

Set default SW_AGENT_MAX_BUFFER_SIZE back to 10000 from 1000, which turns out to be too small for python (many unnecessarily dropped spans).

feature
opened by tom-pytel 14
feat: report metrics related to pvm

Metrics

CPU

total_cpu_utilization

process_cpu_utilization

cpu utilization of the current process

thread_active_count

number of threads invoked by the current process and its children

Memory

total_mem_utilization

process_mem_utilization

memory utilization of the current process

GC

gc_g0

number of objects of generation 0

gc_g1

number of objects of generation 1

gc_g2

number of objects of generation 2

gc_time

For now, MeterReportService is available with gRPC only. So does the e2e test. I will implement MeterReportService and enable e2e test for Kafka later.

Welcome any advice about naming.
feature core

opened by jiang1997 12
[Fix][Plugin] sw_flask general exceptions handled
sw_flask fix will handle errors like returning the wrong type from a handler or other internal errors.

Updated StackedSpan to track depth, and made depth variable instance instead of class level (this was a bug).

Changed how SpanContext decides when all spans finished to write Segment data, now counts span start / stops which should work better across different async scenarios.

Changed new_exit_span() with span.inject() to work simpler like the NodeJS agent, now plugins inject directly themselves if they need to.

Removed carrier from plugins which didn't actually use it.

bug plugin
opened by tom-pytel 12
Add plugin for bottle
[x] Add a test case for the new plugin

[x] Add a CHANGELOG entry for the new plugin

[x] Add a component id in the main repo

[x] Add a logo in the UI repo

[x] Rebuild the requirements.txt by running tools/env/build_requirements_(linux|windows).sh

[x] Rebuild the Plugins.md documentation by running make doc-gen

plugin
opened by jiang1997 11
Enable HTTP log reporting

This is a work in progress.

Now reporting logs in JSON through HTTP to http://oap/v3/logs does work.

But, I'm not sure if the oap/v3/logs endpoint is for such usage(It seems designed for fluent-bit batch reporting?). Reporting logs one by one through HTTP may not be ideal in terms of performance. I'm not sure whether the Java agent only implements gRPC reporter intentionally out of this reason.

Please advise.

Signed-off-by: YihaoChen [email protected]
feature

opened by Superskyyy 11
Feature: collect and report logs
This is a OSPP Summer 2021 project supervised by @Humbertzhang | apache/skywalking#7118

The feature implements optional log reporter functionalities in alignment with the SkyWalking Java agent.

Intercepts logs from Python logging module.

Reports logs via a new temporary gRPC channel(to be removed in the future).

Supports unformatted/ formatted logs with custom layout.

Supports custom logging level threshold.

Bumps up submodule to support log collection protocols.

Bumps up skywalking-eyes and adds a config entry to ignore .gitignore during license checks.

feature core
opened by Superskyyy 11
Introduce better coding style

This PR migrates existing usage of old-style string manipulation (%, .format and +) to f-strings for better clarity and some performance boost.

Adds stricter coding style.

Optimizes debug messages for performance.

Signed-off-by: Superskyyy [email protected]
enhancement chore

opened by Superskyyy 10
Add E2E coverage for Trace and Logging
Change Fastapi to FastAPI component name (sync with OAP change)

Add E2E coverage for Python agent (Trace and Logging), profiling is not tested for now.

Renamed a build variable to remove possible mix-ups with CLI environment Var.

Enhance plugin test to reuse images.

Combined CI to pass if changes are non-essential to agent and workflows.

Note: There's a flaky test due to a possible {{- contains }} issue in the Infra-e2e verifier, the logs seem to be queried in an unstable order by the CLI where later log often placed first in log list, combining with the contains bug, it becomes flaky.

A time.sleep() workaround makes sure the log arrives in clear order which passes the e2e for now.
test
opened by Superskyyy 9
[Enhancement] Async tasks should work 100%

These changes should address all remaining async-related problems not handled in #88. The spans list has been moved from an instance variable in SpanContext to a global task-local variable which allows "forking" it for new subtasks so that multiple concurrent async subtasks don't interfere with one another. spans is now a top-level module var because Python stipulates that ContextVar should be such and only one context exists at any given time so it is essentially a singleton anyway.
enhancement core

opened by tom-pytel 9
Enhance redis plugin to adapt Virtual Cache

Resolves: #10212
enhancement

opened by Jedore 2

Releases(v0.8.0)

v0.8.0(Jul 10, 2022)
What's Changed

spans correctly reference finished parents by @tom-pytel in https://github.com/apache/skywalking-python/pull/161

Refactor current Python agent docs to serve on SkyWalking official website by @Superskyyy in https://github.com/apache/skywalking-python/pull/162

fix broken url by @JaredTan95 in https://github.com/apache/skywalking-python/pull/163

Remove docs from main README.md - Add website doc links. by @Superskyyy in https://github.com/apache/skywalking-python/pull/164

Refactor SkyWalking Python to use the CLI for CI instead of legacy setup by @Superskyyy in https://github.com/apache/skywalking-python/pull/165

Remove places that mentions Python 3.5 support due to EOL by @Superskyyy in https://github.com/apache/skywalking-python/pull/166

Cleanup and Python 3.10 support by @Superskyyy in https://github.com/apache/skywalking-python/pull/167

filled out rest of psycopg 3 plugin by @tom-pytel in https://github.com/apache/skywalking-python/pull/168

Move flake configs all together by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/169

Enable faster CI by categorical parallelism by @Superskyyy in https://github.com/apache/skywalking-python/pull/170

Introduce better coding style by @Superskyyy in https://github.com/apache/skywalking-python/pull/171

Minimize exclude in lint by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/173

Introduce another set of flake8 extensions by @Superskyyy in https://github.com/apache/skywalking-python/pull/174

fix aiohttp outgoing request url by @tom-pytel in https://github.com/apache/skywalking-python/pull/175

bugfix: flask + nginx got KeyError: 'REMOTE_PORT' in sw_flask.py plugin by @VxCoder in https://github.com/apache/skywalking-python/pull/176

Add missing ending quote by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/177

updated deprecated configuration by @doddi in https://github.com/apache/skywalking-python/pull/179

Add plugin for mysqlclient by @katelei6 in https://github.com/apache/skywalking-python/pull/178

Fix sw-rabbitmq TypeError when there are no headers by @dcryans in https://github.com/apache/skywalking-python/pull/182

Fix agent bootup traceback not shown in sw-python CLI by @Superskyyy in https://github.com/apache/skywalking-python/pull/183

Add plugin for FastAPI by @katelei6 in https://github.com/apache/skywalking-python/pull/181

Support CI container logs by @Superskyyy in https://github.com/apache/skywalking-python/pull/185

Update mySQL plugin to support two different parameter keys. by @katelei6 in https://github.com/apache/skywalking-python/pull/186

add the code > 400 is error by @katelei6 in https://github.com/apache/skywalking-python/pull/187

Doc: add how to use with uwsgi by @arcosx in https://github.com/apache/skywalking-python/pull/188

Fix: right doc link for How To Use With uWSGI by @arcosx in https://github.com/apache/skywalking-python/pull/189

The local log stack depth is not truncated by @wzy960520 in https://github.com/apache/skywalking-python/pull/190

Revert "The local log stack depth is not truncated (#190)" by @Superskyyy in https://github.com/apache/skywalking-python/pull/191

Fix sw_formatter wrongly set cache that impacts user handlers by @Superskyyy in https://github.com/apache/skywalking-python/pull/192

Fix typo that cause failure in loading user sitecustomize.py by @Superskyyy in https://github.com/apache/skywalking-python/pull/193

Drop support for Flask 1.x due to EOL & Jinja2 issue by @Superskyyy in https://github.com/apache/skywalking-python/pull/195

Update agent docs, changelogs and PR template. by @Superskyyy in https://github.com/apache/skywalking-python/pull/197

Fix Python shown as UNKNOWN language in OAP by @Superskyyy in https://github.com/apache/skywalking-python/pull/194

Fix logging level not properly set according to config by @Superskyyy in https://github.com/apache/skywalking-python/pull/196

Fix the properties are not set correctly by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/198

Add E2E coverage for Trace and Logging by @Superskyyy in https://github.com/apache/skywalking-python/pull/199

Support log reporter safe mode by @Superskyyy in https://github.com/apache/skywalking-python/pull/200

Fix scheduled event fails changes CI by @Superskyyy in https://github.com/apache/skywalking-python/pull/201

Fix deadlink and CI on schedule by @Superskyyy in https://github.com/apache/skywalking-python/pull/203

Remove namespace to instance properties and add pid by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/205

Enhance Traceback depth default to 10 by @Superskyyy in https://github.com/apache/skywalking-python/pull/206

Unify the tag name with other agents by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/208

Improved ignore path regex by @tom-pytel in https://github.com/apache/skywalking-python/pull/210

Fix sw_psycopg2 register_type() by @tom-pytel in https://github.com/apache/skywalking-python/pull/211

Fix psycopg2 register_type() second arg default by @tom-pytel in https://github.com/apache/skywalking-python/pull/212

Add plugin doc check in CI by @JarvisG495 in https://github.com/apache/skywalking-python/pull/213

Fix typo in docker/Makefile by @jiang1997 in https://github.com/apache/skywalking-python/pull/216

Update UI repository link in PR template by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/217

Add plugin for bottle by @jiang1997 in https://github.com/apache/skywalking-python/pull/214

Draft release 0.8.0 by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/219

Migrate license checker to license-eye and adjust the release script by @kezhenxu94 in https://github.com/apache/skywalking-python/pull/220

New Contributors

@JaredTan95 made their first contribution in https://github.com/apache/skywalking-python/pull/163

@VxCoder made their first contribution in https://github.com/apache/skywalking-python/pull/176

@doddi made their first contribution in https://github.com/apache/skywalking-python/pull/179

@katelei6 made their first contribution in https://github.com/apache/skywalking-python/pull/178

@dcryans made their first contribution in https://github.com/apache/skywalking-python/pull/182

@arcosx made their first contribution in https://github.com/apache/skywalking-python/pull/188

@wzy960520 made their first contribution in https://github.com/apache/skywalking-python/pull/190

@JarvisG495 made their first contribution in https://github.com/apache/skywalking-python/pull/213

@jiang1997 made their first contribution in https://github.com/apache/skywalking-python/pull/216

Full Changelog: https://github.com/apache/skywalking-python/compare/v0.7.0...v0.8.0
Source code(tar.gz)
Source code(zip)
v0.7.0(Sep 13, 2021)
Feature:

Support collecting and reporting logs to backend (#147)

Support profiling Python method level performance (#127

Add a new sw-python CLI that enables agent non-intrusive integration (#156)

Add exponential reconnection backoff strategy when OAP is down (#157)

Support ignoring traces by http method (#143)

NoopSpan on queue full, propagation downstream (#141)

Support agent namespace. (#126)

Support secure connection option for GRPC and HTTP (#134)

Plugins:

Add Falcon Plugin (#146)

Update sw_pymongo.py to be compatible with cluster mode (#150)

Add Python celery plugin (#125)

Support tornado5+ and tornado6+ (#119)

Fixes:

Remove HTTP basic auth credentials from log, stacktrace, segment (#152)

Fix @trace decorator not work (#136)

Fix grpc disconnect, add SW_AGENT_MAX_BUFFER_SIZE to control buffer queue size (#138)

Others:

Chore: bump up requests version to avoid license issue (#142)

Fix module wrapt as normal install dependency (#123)

Explicit component inheritance (#132)

Provide dockerfile & images for easy integration in containerized scenarios (#159)

Source code(tar.gz)
Source code(zip)
v0.6.0(Mar 31, 2021)
Fixes:

Segment data loss when gRPC timing out. (#116)

sw_tornado plugin async handler status set correctly. (#115)

sw_pymysql error when connection haven't db. (#113)

Source code(tar.gz)
Source code(zip)
v0.5.0(Dec 31, 2020)
New plugins

Pyramid Plugin (#102)

AioHttp Plugin (#101)

Sanic Plugin (#91)

API and enhancements

@trace decorator supports async functions

Supports async task context

Optimized path trace ignore

Moved exception check to Span.__exit__

Moved Method & Url tags before requests

Fixes:

BaseExceptions not recorded as errors

Allow pending data to send before exit

sw_flask general exceptions handled

Make skywalking logging Non-global

Chores and tests

Make tests really run on specified Python version

Deprecate 3.5 as it's EOL

Source code(tar.gz)
Source code(zip)
v0.4.0(Nov 24, 2020)
Feature: Support Kafka reporter protocol (#74)

BugFix: Move generated packages into skywalking namespace to avoid conflicts (#72)

BugFix: Agent cannot reconnect after server is down (#79)

Test: Mitigate unsafe yaml loading (#76)

Source code(tar.gz)
Source code(zip)
v0.3.0(Aug 28, 2020)
New plugins

Urllib3 Plugin (#69)

Elasticsearch Plugin (#64)

PyMongo Plugin (#60)

Rabbitmq Plugin (#53)

Make plugin compatible with Django (#52)

API

Add process propagation (#67)

Add tags to decorators (#65)

Add Check version of packages when install plugins (#63)

Add thread propagation (#62)

Add trace ignore (#59)

Support snapshot context (#56)

Support correlation context (#55)

Chores and tests

Test: run multiple versions of supported libraries (#66)

Chore: add pull request template for plugin (#61)

Chore: add dev doc and reorganize the structure (#58)

Test: update test health check (#57)

Chore: add make goal to package release tar ball (#54)

Source code(tar.gz)
Source code(zip)
v0.2.0(Jul 28, 2020)
New plugins

Kafka Plugin (#50)

Tornado Plugin (#48)

Redis Plugin (#44)

Django Plugin (#37)

PyMsql Plugin (#35)

Flask plugin (#31)

API

Add ignore_suffix Config (#40)

Add missing log method and simplify test codes (#34)

Add content equality of SegmentRef (#30)

Validate carrier before using it (#29)

Chores and tests

Test: print the diff list when validation failed (#46)

Created venv builders for linux/windows and req flashers + use documentation (#38)

Source code(tar.gz)
Source code(zip)
v0.1.0(Jun 28, 2020)
API: agent core APIs, check the APIs and the examples

Plugin: built-in libraries http, urllib.request and third-party library requests are supported.

Test: agent test framework is setup, and the corresponding tests of aforementioned plugins are also added.

Source code(tar.gz)
Source code(zip)

Owner

The Apache Software Foundation

GitHub Repository https://skywalking.apache.org/

Minos-python - A framework which helps you create reactive microservices in Python

minos-python Summary [TODO] Packages minos-microservice-aggregate minos-microser

380 Jan 04, 2023

Ningyu Jia(nj2459)/Mengyin Ma(mm5937) Call Analysis group project(Group 36)

Group and Section Group 36 Section 001 name and UNI Name UNI Ningyu Jia nj2459 Mengyin Ma mm5937 code explanation Parking.py (1) Calculate the rate of

1 Dec 04, 2021

Pyrmanent - Make all your classes permanent in a flash 💾

Pyrmanent A base class to make your Python classes permanent in a flash. Features Easy to use. Great compatibility. No database needed. Ask for new fe

4 Jan 07, 2022

Fork of pathlib aiming to support the full stdlib Python API.

pathlib2 Fork of pathlib aiming to support the full stdlib Python API. The old pathlib module on bitbucket is in bugfix-only mode. The goal of pathlib

73 Dec 23, 2022

Repository for my Monika Assistant project

Monika_Assistant Repository for my Monika Assistant project Major changes: Added face tracker Added manual daily log to see how long it takes me to fi

3 Jan 10, 2022

Render reMarkable documents to PDF

rmrl: reMarkable Rendering Library rmrl is a Python library for rendering reMarkable documents to PDF files. It takes the original PDF document and th

95 Dec 25, 2022

A flexible free and unlimited python tool to translate between different languages in a simple way using multiple translators.

deep-translator Translation for humans A flexible FREE and UNLIMITED tool to translate between different languages in a simple way using multiple tran

806 Jan 04, 2023

Integration of Hotwire's Turbo library with Flask.

turbo-flask Integration of Hotwire's Turbo library with Flask, to allow you to create applications that look and feel like single-page apps without us

240 Jan 06, 2023

This python code will get requests from SET (The Stock Exchange of Thailand) a previously-close stock price and return it in Thai Baht currency using beautiful soup 4 HTML scrapper.

1 Oct 24, 2022

Example of my qtile config using the gruvbox colorscheme.

QTILE config Example of my qtile config using the gruvbox colorscheme. unicodes.py unicodes.py returns a widget.TextBox with a unicode. Currently it c

31 Jan 02, 2023

This is a simple bank management system based on Python.

Python Bank Management This is a simple bank management system based on Python. It's able to do basic operations of simple bank management. Outcome: W

3 Jan 06, 2022

Insights in greek football league 2020-2021 and bookmaker's accuracy

Greek_Football_League_Analysis_2020_2021 Aim of Project: This project aims in deriving useful insights from greek football league 2020-2021 by mean st

2 Jan 16, 2022

Council Data Project is an open-source project dedicated to providing journalists, activists, researchers, and all members of each community we serve with the tools they need to stay informed and hold their Council Members accountable.

CDP - Self Council Data Project Council Data Project is an open-source project dedicated to providing journalists, activists, researchers, and all mem

1 May 12, 2022

The Python agent for Apache SkyWalking

Related tags

Overview

SkyWalking Python Agent

Documentation

Installation Requirements

Contributing

Contact Us

License

Comments

Behavior

Syntactic sugar

Counter

Histogram

Simple test

Metrics

CPU

total_cpu_utilization

process_cpu_utilization

thread_active_count

Memory

total_mem_utilization

process_mem_utilization

GC

gc_g0

gc_g1

gc_g2

gc_time

Releases(v0.8.0)

v0.8.0(Jul 10, 2022)

What's Changed

New Contributors

v0.7.0(Sep 13, 2021)

v0.6.0(Mar 31, 2021)

v0.5.0(Dec 31, 2020)

v0.4.0(Nov 24, 2020)

v0.3.0(Aug 28, 2020)

v0.2.0(Jul 28, 2020)

v0.1.0(Jun 28, 2020)

Owner

The Apache Software Foundation

Minos-python - A framework which helps you create reactive microservices in Python

Ningyu Jia(nj2459)/Mengyin Ma(mm5937) Call Analysis group project(Group 36)

Pyrmanent - Make all your classes permanent in a flash 💾

Fork of pathlib aiming to support the full stdlib Python API.

Repository for my Monika Assistant project

Render reMarkable documents to PDF

A flexible free and unlimited python tool to translate between different languages in a simple way using multiple translators.

Integration of Hotwire's Turbo library with Flask.

This python code will get requests from SET (The Stock Exchange of Thailand) a previously-close stock price and return it in Thai Baht currency using beautiful soup 4 HTML scrapper.

Example of my qtile config using the gruvbox colorscheme.

This is a simple bank management system based on Python.

Insights in greek football league 2020-2021 and bookmaker's accuracy

Council Data Project is an open-source project dedicated to providing journalists, activists, researchers, and all members of each community we serve with the tools they need to stay informed and hold their Council Members accountable.

My collection of mini-projects in various languages

Repo created for the purpose of adding any kind of programs and projects

"Cambio de monedas" Change-making problem with Python, dynamic programming best solutions,

Calibre Libgen Non-fiction / Sci-tech store plugin

An educational platform for students

Logging-monitoring-instrumentation - A brief repository on logging monitoring and instrumentation in Python

TurtleBot Control App - TurtleBot Control App With Python