Pyrite

Developer Guide

This document describes the internal structure of the MPI communication tracer, trace consumer, analytics functionality, and visualisation functionality. This includes the interceptor library, parser, analytics pipeline, frontend display architecture, and test suite.

1. Overview

The project has three main runtime stages:

  1. MPI interception
    • PMPI wrappers record MPI activity to a binary .mpic trace.
  2. Parsing and analysis
    • tools/mpi_data_parser.py converts .mpic into .mpix.
    • It also extracts summary analytics and heuristic performance findings.
  3. Frontend visualisation
    • vis/ loads .mpix, streams timeline chunks on demand, renders hardware and traffic in 3D, and overlays analytics.

2. Source Tree

src/
  mpi_communication_tracking.c
  mpi_communication_tracking.h

tools/
  mpi_data_parser.py
  topology_generator.py
  slurm_topology_generator.py

vis/
  index.html
  style.css
  visualiser.js
  analytics.js
  analytics-3d.js
  analytics-controls.js

tests/
  CMakeLists.txt
  ctest_driver.py
  trace_parser.py
  test_*.c
  test_*.f90

3. Interceptor Library

3.1 Purpose

The interceptor is a shared library loaded with LD_PRELOAD. It wraps MPI calls via PMPI and records trace records with low overhead.

3.2 Core responsibilities

3.3 Trace record types

The library writes two record classes:

3.4 Supported calls

Point-to-point

Completion and sync

Collectives

3.5 Nonblocking tracking

The interceptor also tracks pending non-blocking requests internally. Instead of logging nonblocking communication only at posting time, it:

This allows:

3.6 Fortran wrappers

The library also includes symbol-level Fortran wrappers, primarily intended for:

Coverage includes non-blocking completions and request-handling routines, but it is not intended as a full portable mpi_f08 interception layer yet.

4. .mpic Binary Format

The .mpic file is the raw output of the interceptor.

4.1 Global header

The file begins with:

4.2 Process info records

One per rank, containing:

4.3 Per-rank sections

For each rank:

  1. rank id
  2. "P2P Small Type Messages"
  3. number of small records
  4. small records
  5. "P2P Large Type Messages"
  6. number of large records
  7. large records

4.4 Parser strategy

To process a trace file we provide a tool; tools/mpi_data_parser.py, which is designed to try and parse the file as created, but can also deal with malformed or truncated files, using the following approach:

  1. strict parsing
    • trusts section counts from the trace file
    • detects malformed structure early
  2. salvage parsing
    • scans for section anchors
    • useful for partially damaged files

This is provided to try and enable some profile/tracing data to be analysed if in a program run fails to complete successfully, but the primary mode of working requires a correctly formed trace file.

5. Parser and Analysis Pipeline

5.1 Purpose

tools/mpi_data_parser.py converts .mpic into .mpix and creates a visualisation-friendly analysis layer.

5.2 Outputs

The parser emits a compressed .mpix container with:

5.3 Timeline chunking

The timeline is split into fixed-size event chunks and compressed with zlib. This allows the visualisation tool to only loads only the chunks needed for the current time window, enabling large traces to be visualised without requiring the full data to be in memory at the same time.

6. Analytics Model

The parser constructs an analysis object in the .mpix header.

6.1 High-level structure

{
  "summary": { ... },
  "per_rank": [ ... ],
  "top_ranks_by_out_bytes": [ ... ],
  "top_ranks_by_in_bytes": [ ... ],
  "top_ranks_by_touch_bytes": [ ... ],
  "top_links": [ ... ],
  "collective_roots": [ ... ],
  "barrier_spreads": [ ... ],
  "patterns": [ ... ],
  "issues": [ ... ],
  "time_windows": [ ... ]
}

6.2 Summary

Includes:

6.3 Per-rank summaries

Includes:

Hottest sender/receiver links from the canonical transfer subset.

6.5 Collective roots

Summaries for concentrated rooted collective traffic.

6.6 Barrier spreads

Heuristic skew estimates based on repeated barrier timing order.

6.7 Time windows

Coarse phase summaries, including per-window:

7. Pattern Detection

The parser currently infers several communication motifs heuristically.

7.1 Master/worker or star-like behaviour

Triggered when one rank dominates communication volume and degree.

7.2 Rooted collective concentration

Triggered when rooted collective traffic is concentrated on a small number of roots.

7.3 Ring / nearest-neighbour

Triggered when traffic is dominated by short rank-distance communication.

7.4 Neighbourhood exchange / halo-like behaviour

Triggered when communication concentrates on a small set of offsets and reciprocal pairs.

7.5 All-to-all-like behaviour

Triggered when the pair graph is dense and peer counts are high.

7.6 Ping-pong pairs

Triggered when balanced two-way communication is detected between a small number of pairs.

8. Issue Detection

The parser emits heuristic issue records for likely performance problems.

Current issue types include:

Each issue may contain:

Important: these are diagnostics for visualisation and first-pass investigation, not formal proof of a problem.

9. Frontend Architecture

The frontend is split into several files.

9.1 visualiser.js

Main application logic:

9.2 analytics.js

Renders the analytics panel:

Also supports:

9.3 analytics-3d.js

Renders persistent 3D analytics overlays:

Also supports:

9.4 analytics-controls.js

Renders:

10. Frontend Data Flow

10.1 File loading

index.html loads .mpix through the browser file picker.

10.2 Header load

The frontend reads the compressed header first and populates:

10.3 Timeline chunk load

As playback advances, the frontend loads only the relevant chunk using the offsets stored in the header.

10.4 Live rendering

At each playback step, the frontend:

10.5 Analytics overlays

These are independent of playback-time communication rendering and are built from parsedData.analysis.

11. 3D Analytics Overlay Behaviour

11.1 Persistent overlays

These remain visible even when playback is paused or the current active timeline has moved elsewhere.

11.2 Isolation

When the user clicks Isolate 3D on an issue card:

11.3 Layout changes

After layout changes, overlays are refreshed so arcs and halos align with the new positions.

12. Testing

The project uses CTest-native integration tests.

12.1 C tests

Coverage includes:

12.2 Fortran tests

Optional Fortran tests cover:

Configure Fortran support

cmake -S . -B build -DMPI_TRACE_FORTRAN_TESTS=AUTO
cmake -S . -B build -DMPI_TRACE_FORTRAN_TESTS=ON
cmake -S . -B build -DMPI_TRACE_FORTRAN_TESTS=OFF

Run tests

cd build
ctest --output-on-failure

13. Extending the Interceptor

Common extension points in src/ include:

Add a new MPI wrapper

  1. define a message type id in the header
  2. add the C wrapper
  3. decide whether it maps to:
    • small record
    • large record
    • request-tracked completion behaviour
  4. extend parser message type tables
  5. extend visualiser category tables
  6. add tests

Add new Fortran coverage

  1. add symbol-layer wrapper(s)
  2. convert handles with MPI_Fint conversion routines
  3. convert statuses carefully
  4. add CTest Fortran coverage

14. Extending the Parser

Common extensions in tools/mpi_data_parser.py:

Add new MPI message types

Add new analytics

Design guidance

Analytics should be:

15. Extending the Frontend

Add new analytics cards

Use analytics.js and follow the existing card helper pattern.

Add new persistent overlays

Use analytics-3d.js and:

Add new controls

Use analytics-controls.js and wire them to Analytics3D.configure(...).

16. Current Limitations

17. Potential Future Work

Potential next improvements include:

18. Typical Development Workflow

# Configure
cmake -S . -B build -DMPI_TRACE_FORTRAN_TESTS=AUTO

# Build
cmake --build build

# Run tests
ctest --test-dir build --output-on-failure

# Profile an application
LD_PRELOAD=$PWD/build/src/libmpi_comm_tracker.so mpirun -n 16 ./your_mpi_application

# Parse trace
python tools/parse_mpic.py your_mpi_application-YYYYMMDDHHMMSS.mpic hardware_map.json

# Open frontend
# Load vis/index.html in a browser and open the .mpix file

19. Notes for Maintainers

When changing one layer, remember the others:

A safe workflow is:

  1. update or add tests
  2. update interceptor/parser/frontend
  3. re-run CTest
  4. validate with a known .mpix trace in the browser
This site is open source. Improve this page.