find-cpython-usage

Find CPython Internal Usage Skill

This skill helps identify all CPython internal headers and structures used in the codebase, which is essential when adding support for new Python versions.

When to Use This Skill

Use this skill when:

• Adding support for a new Python version
• Investigating CPython API dependencies
• Understanding what internal APIs the profiler uses
• Preparing to compare CPython versions

Key Principles

1. Focus on internal headers - These are most likely to change between versions
2. Check all native extensions - CPython internals are used in profiling, AppSec, and internal modules
3. Look for struct field access - Direct field access is version-sensitive
4. Document findings - Keep track of what you find for comparison

How This Skill Works

Step 1: Find CPython Header Includes

Search for CPython header includes across all C/C++/Cython files:

# Find all CPython internal header includes
grep -r "include.*internal/pycore" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx"

# Find all CPython cpython header includes
grep -r "include.*cpython" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx"

# Find all Python.h includes (indicates CPython API usage)
grep -r "#include.*Python\.h" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx"

# Find frameobject.h includes (common CPython API)
grep -r "#include.*frameobject\.h" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx"

# Find PyO3 FFI usage in Rust (Rust extensions may use CPython internals)
grep -r "pyo3_ffi::\|pyo3::ffi::" src/native/ --include="*.rs" || true

Note: These patterns search across all native extension files (.c, .cpp, .h, .hpp, .pyx, .rs) regardless of their location in the codebase. Check setup.py to see which extensions are built. Rust extensions use PyO3 which may access CPython internals through the pyo3_ffi module.

Step 2: Find Struct Field Access

Search for direct struct field access and struct definitions across all native files:

# Find struct field accesses (arrow operator)
grep -r "->f_\|->[a-z_]*\." ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp"

# Find struct definitions
grep -r "struct.*Py" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp"

# Find common CPython struct usage
grep -r "PyFrameObject\|PyThreadState\|_PyInterpreterFrame" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx"
grep -r "PyFrameObject\|PyThreadState\|PyInterpreterState" src/native/ --include="*.rs" || true

# Find PyCodeObject usage
grep -r "PyCodeObject" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx"
grep -r "PyCodeObject" src/native/ --include="*.rs" || true

Look for patterns like:

• Frame structure access (PyFrameObject, _PyInterpreterFrame)
• Thread state access (PyThreadState)
• Code object access (PyCodeObject)
• Generator/coroutine structures
• Asyncio task structures

Step 3: Identify Common Structures

Common CPython structures we typically access:

Frame structures:

• PyFrameObject / struct _frame
• _PyInterpreterFrame

State structures:

• PyThreadState
• PyInterpreterState
• _PyRuntimeState

Code structures:

• PyCodeObject

Generator structures:

• PyGenObject
• PyAsyncGenASend

Asyncio structures:

• FutureObj
• TaskObj

Step 4: Document Findings

Create a list of:

• All headers that are included
• All structs that are accessed
• All struct fields that are used directly

This will be used in the next step to compare against the new Python version.

Native Extensions Using CPython APIs

To find all native extensions that may use CPython APIs, check setup.py:

# View all native extensions defined in setup.py
grep -A 5 "Extension\|CMakeExtension\|Cython.Distutils.Extension\|RustExtension" setup.py

The setup.py file defines all native extensions (C, C++, CMake, Cython, and Rust) that are built for the project. Not all extensions use CPython internals - focus on those that access frame objects, thread state, or internal structures when searching for CPython API usage.

Note: Rust extensions use PyO3 bindings which may access CPython internals through pyo3_ffi module. Search Rust source files (.rs) for CPython API usage as well.

Common Headers to Look For

The grep commands above will identify which CPython headers are actually used in the codebase. Common patterns include:

Public Headers:

• Headers matching *.h in Include/ directory (e.g., frameobject.h, unicodeobject.h)
• Headers in Include/cpython/ directory (e.g., cpython/genobject.h)

Internal Headers (require Py_BUILD_CORE):

• Headers matching internal/pycore*.h pattern
• These are most likely to change between Python versions

Focus on headers that are actually found by the grep commands rather than maintaining a hardcoded list, as the headers used may change over time.

Output Format

After running this skill, you should have:

1. A list of all CPython headers included in the codebase
2. A list of all CPython structs accessed
3. A list of struct fields accessed directly
4. Files that use each header/struct

This information can then be used with the compare-cpython-versions skill to identify what changed.

Related

• compare-cpython-versions skill: Use findings from this skill to compare versions