ansible-trace

Visualise where time is spent in your Ansible playbooks: what tasks, and what hosts, so you can find where to optimise and decrease playbook latency.

An Ansible Callback Function which traces the execution time of Ansible playooks, outputting Chrome's Trace Event Format for visualising in the Perfetto in-browser trace UI.

Here's a trace of me deploying to my home Raspberry Pi cluster, with the default strategy: linear. You can see that now all the tasks are synchronized across hosts, with each host waiting for the slowest host before proceeding to the next task:

Here's the same playbook ran with strategy: free so fast hosts run to completion without waiting for slow hosts:

You can click on tasks to see details about them:

Interactive Example

1. Download (Right-click -> Save Link As) example-trace.json.
2. Open https://ui.perfetto.dev/, and drag and drop in the downloaded example-trace.json.

Usage

1. Copy trace.py into your Ansible's callback_plugins directory, or in other positions Ansible accepts, e.g.:

   ansible-root
   ├── ansible.cfg
   ├── site.yml
   └── callback_plugins
       └── trace.py
   

2. Enable the trace callback plugin in your ansible.cfg:

   [defaults]
   callback_enabled = trace

   Or, enable it at the top of your playbook yml:

   ansible:
     env:
       CALLBACKS_ENABLED: trace
       TRACE_OUTPUT_DIR: .
       TRACE_HIDE_TASK_ARGUMENTS: True

3. Run your Ansible Playbook:

   $ ansible-playbook site.yml

   This will output trace.json in the TRACE_OUTPUT_DIR, defaulting to your current working directory.

4. Open https://ui.perfetto.dev/, and drag-and-drop in the trace.json.

   You don't have to wait for the trace to finish; you can open in-progress trace files.

Other Trace Viewers

Perfetto is the most mature trace viewer, but here are some other options:

• chrome://tracing (aka Catapult Trace Viewer) is the older version of Perfetto. Supports generating a standalone HTML page.
• Speedscope can open the traces, but only shows one host at a time.
• Firefox Profiler can open the traces, showing trace spans in the "Marker Chart" tab: example.

Other Ansible Profiling Tools

profile_tasks

ansible.posix.profile_tasks displays task timing as console output, but can't visualise gaps in the timing (e.g. with strategy: linear when fast hosts wait for slow hosts).

ansible-playbook -vvvv

Adding extra vs adds more debug info, -vvvv enables connection debugging.

Mitogen for Ansible

Mitogen promises to speed up your Ansible playbooks with a persistent interpreter. They profile their runs for bandwidth an time by analysing network packet captures.

You need to install from HEAD to support latest Ansible versions, because there hasn't been a tagged release since 2019.