1 capture
01 Jun 2025
May JUN Jul
01
2024 2025 2026
success
fail

About this capture

COLLECTED BY

Collection: Save Page Now

TIMESTAMPS

The Wayback Machine - http://web.archive.org/web/20250601114607/https://github.com/python/cpython/pull/129337
 

Skip to content  

Navigation Menu

 






Sign in  
Appearance settings  
from  







Draft  

gh-132502: Prototyping adding timestamps to tracebacks.   #129337


 wants to merge  55 commits into   from  









+1,763   165  




Conversation



This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.  Learn more about bidirectional Unicode characters  
Show hidden characters







Copy link  

Member  


@gpshead gpshead   commented  Jan 27, 2025    






The idea for this came up at work with @njsmith tossing the idea out as "dumb" yet still maybe useful... Why? to make debugging what happened in async servers with lots of exception groups and exceptions easier. Event timestamps reported on everything when emitting exception groups containing tracebacks often with their own nested causes would allow some semblance of order to be more readily understood and correlated with other parts of the distributed system.

This draft PR is a demo. If we want such a feature, we should settle on semantics in a Discuss thread and if necessary, write it up as a PEP. This should be simpler than exception notes (PEP-678) was. One thought was just to store the timestamp as a note; but that'd involve string and list creation on every exception - slow.

Demo


>> def california_raisin(): ... try: ... raise RuntimeError("not enough sunshine") ... except Exception: ... raise OSError(2, "on a cloudy day") ... >>> california_raisin() Traceback (most recent call last):  File "", line 3, in california_raisin  raise RuntimeError("not enough sunshine") RuntimeError: not enough sunshine <@2025-02-01T20‥43:01.026169> During handling of the above exception, another exception occurred: Traceback (most recent call last):  File "", line 1, in  california_raisin()  ~~~~~~~~~~~~~~~~~^^  File "", line 5, in california_raisin  raise OSError(2, "on a cloudy day") FileNotFoundError: [Errno 2] on a cloudy day <@2025-02-01T20‥43:01.026176Z> >>>">
± PYTHON_TRACEBACK_TIMESTAMPS=iso ../b/python
Python 3.14.0a4+ (heads/traceback-timestamps-dirty:b06539429ef, Feb  1 2025, 17:54:53) [Clang 18.1.3 (1ubuntu1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def california_raisin():
...     try:
...         raise RuntimeError("not enough sunshine")
...     except Exception:
...         raise OSError(2, "on a cloudy day")
...
>>> california_raisin()
Traceback (most recent call last):
  File "<python-input-0>", line 3, in california_raisin
    raise RuntimeError("not enough sunshine")
RuntimeError: not enough sunshine <@2025-02-01T20:43:01.026169>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<python-input-2>", line 1, in <module>
    california_raisin()
    ~~~~~~~~~~~~~~~~~^^
  File "<python-input-0>", line 5, in california_raisin
    raise OSError(2, "on a cloudy day")
FileNotFoundError: [Errno 2] on a cloudy day <@2025-02-01T20:43:01.026176Z>
>>>


A real world demo involving exception groups in an async application would be more interesting but isn't something I can just tap out in a repl...

Updates: [edits]


I've done performance testing. It now appears to be a no-op (no meaningful regression) as desired performance wise on Linux. For that to be true, I had to add the one special case I suspected might matter: Don't collect the timestamp on StopIteration and AsyncStopIteration as those are not infrequent error exceptions, but a normal part of application control flow. Without that, one of the async pyperformance benchmarks showed a significant performance hit of over 10%.

I've changed it to not emit the timestamps by default. If you set the PYTHON_TRACEBACK_TIMESTAMPS= environment variable to one of either usor1, ns, or iso, timestamps will be enabled in traceback module display output. See the documentation in the PR.

TODO



 Fix the reference leak crash in iso mode. Or decide iso is too complicated and simplify by ripping it out?

 macOS performance regression testing, on both architectures.

 Windows performance regression testing.

 Fix the 32-bit build test_sys exception struct size failures.

Tests needed:

 I doubt the existing test suite passes if that environment variable is set?

 Specifically test pickle and unpickle of all existing exception types in our heirarchy.

 Test user derived class exception types from each of BaseException, Exception, OSError, ImportError, AttributeError to have the expected attributes and to survive pickle dump and load with the timestamp intact.

 Test that the timestamp shows up when enabled on everything but StopIteration and AsyncStopIteration.



 Can I have a PONY?



Issue: Collect timestamps on most exceptions for optional display on tracebacks. #132502

 





asvetlov, tomasr8, bswck, dolfinus, and lemorage reacted with thumbs up emoji










gpshead   added 24commits  January 27, 2025 00:27











Proof of concept: Add timestamps to tracebacks.    






c5a9ccd  



This came up at work as a suggestion to make debugging what happened in
big async servers with lots of exception groups and exceptions easier.
Timestamps when emitting exception groups containing tracebacks often
with their own nested causes would allow some semblance of order to be
understood.

This is a demo.  If we want such a feature, we should settle on
semantics in a Discuss thread and write it up as a PEP.  This should be
simpler than exception notes (PEP-678) was.  One thought was just to
store the timestamp as a note; but that'd involve string and list
creation on every exception.

Performance testing needs to be done. This is the kind of thing that is
visually distracting, so not all applications want to _see_ the
timestamps.  A knob to turn that on for those who do seems more useful
rather than making that the default.  But the performance impact of
merely collecting the timestamps is worth knowing.












PYTHON_TRACEBACK_TIMESTAMPS=1 required to enable their display.  






3dc00c2  













skip timestamp on StopIteration - not an error    






a05766f  



Avoids a 15% regression in the pyperformance async_generators suite.












include the timestamp in exception pickles.  






cdb67f0  













os.environ not os.environb  






99ffc8a  













Cleaner struct layout.  






c119a02  













Timestamp format configurability.  






bcc720b  













Plumb into exception subtypes; including pickling.  






b065394  













minor cleanups  






09a547a  













initial pass at documentation.  






daa752d  













Fix doc references?  






e7fab86  













proper refcount cleanup in error cases.  






e8a6297  













Merge branch 'main' into traceback-timestamps  






6809426  













Allow PyErr_Display* to emit timestamps as well.  






0d83447  













Make the testsuite pass with timestamps turned on.    






7c83ebf  



Tested with `PYTHON_TRACEBACK_TIMESTAMPS=ns` set.

First pass.  Further review could rework some of these changes.

Explicit tests for the new feature have yet to be added.












Docs docs docs docs docs  






beadfb8  













docs make check  






354c5f0  













docs typo  






33d20dd  













GIT  






2f72323  













docs: reword exceptions  






7f7357d  













vi PEBKAC reformatted traceback.rst? undo  






d9d2b1f  













more formatting messup undo  






98b4593  













more undo  






c9ad56d  













reword some docs, add examples  






53b5500  











@gpshead gpshead  force-pushed  the   traceback-timestamps   branch  from  6c85054to53b5500  Compare   February 10, 2025 08:04 







gpshead   added 5commits  February 10, 2025 08:25











REDO BEFORE MERGE: enable on some CI builds  






8043b80  













Go full on -X traceback_timestamps command line flag.    






75072cb  



Better docs, improved tests.

Claude Code using Sonnet 3.7 helped with this, but that was a bit of a battle
as our CPython code context size for this type of change is huge.












config.field && field[0] check. If needed, we could cache the "collect or not" bool in a static global as it is fair to say this is a process wide setting rather than per interpreter, but ugh." data-pjax="true" class="Link--secondary markdown-title" href="/web/20250601114607/https://github.com/python/cpython/pull/129337/commits/eebec1d09b272fb0a3e564ac6a70f19990bbdf13">Disable time collection when display is disabled.    






eebec1d  



TODO: performance testing - this increases the conditional load on every
BaseException instantiation with that interp->config.field && field[0] check.

If needed, we could cache the "collect or not" bool in a static global as it is
fair to say this is a process wide setting rather than per interpreter, but ugh.












remove errant Doc temp db file  






9ff3234  













WIP: optional config string  






f30c74d  

















Copy link  

Member   Author  


gpshead   commented  Apr 14, 2025  



!buildbot .rasp.



















Copy link  



bedevere-bot   commented  Apr 14, 2025  



🤖 New build scheduled with the buildbot fleet by @gpshead for commit 964bdd3 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F129337%2Fmerge

The command will test the builders whose names match following regular expression: .*rasp.*

The builders matched are:

  • ARM Raspbian PR















@gpshead gpshead  added  the  🔨 test-with-refleak-buildbots  Test PR w/ refleak buildbots; report in status section  label  Apr 14, 2025  









Copy link  



bedevere-bot   commented  Apr 14, 2025  



🤖 New build scheduled with the buildbot fleet by @gpshead for commit 964bdd3 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F129337%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.
















@bedevere-bot bedevere-bot  removed  the  🔨 test-with-refleak-buildbots  Test PR w/ refleak buildbots; report in status section  label  Apr 14, 2025  







gpshead   added 3commits  April 13, 2025 20:48











waste time dealing with bad unicode test env. life  






5554b67  













alignment SCIENCE  






0f9bb1b  













errors="ignore" chicanery decode  






a55c3b1  











@gpshead gpshead  added  the  🔨 test-with-refleak-buildbots  Test PR w/ refleak buildbots; report in status section  label  Apr 14, 2025  









Copy link  



bedevere-bot   commented  Apr 14, 2025  



🤖 New build scheduled with the buildbot fleet by @gpshead for commit a55c3b1 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F129337%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.
















@bedevere-bot bedevere-bot  removed  the  🔨 test-with-refleak-buildbots  Test PR w/ refleak buildbots; report in status section  label  Apr 14, 2025  







gpshead   and others  added 3commits  April 13, 2025 22:32











" data-pjax="true" class="Link--secondary markdown-title" href="/web/20250601114607/https://github.com/python/cpython/pull/129337/commits/d2d758311d5390187cd2b253701932c2592e3e79">Fix traceback_timestamps tests for cross-platform line endings    






d2d7583  



Use regex assertions with multi-line mode to match end of lines instead of
literal newline characters. This allows the tests to pass on Windows which
uses \r\n line endings instead of Unix \n line endings.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>












(°°)  Windows  






84989bb  













pedantic-NotIn-too  






b268c82  











@gpshead gpshead  added  the  🔨 test-with-refleak-buildbots  Test PR w/ refleak buildbots; report in status section  label  Apr 14, 2025  









Copy link  



bedevere-bot   commented  Apr 14, 2025  



🤖 New build scheduled with the buildbot fleet by @gpshead for commit b268c82 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F129337%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.
















@bedevere-bot bedevere-bot  removed  the  🔨 test-with-refleak-buildbots  Test PR w/ refleak buildbots; report in status section  label  Apr 14, 2025  





@gpshead gpshead  mentioned this pull request   Apr 14, 2025  


Collect timestamps on most exceptions for optional display on tracebacks.  #132502  

Open  






@gpshead gpshead  changed the title Proof of concept: Add timestamps to tracebacks. gh-132502: Prototyping adding timestamps to tracebacks.  Apr 14, 2025  







gpshead   and others  added 5commits  May 24, 2025 05:08











" data-pjax="true" class="Link--secondary markdown-title" href="/web/20250601114607/https://github.com/python/cpython/pull/129337/commits/d0f95ba6d2023c33f527fb4c8a4259a2a6dc8704">Merge origin/main into traceback-timestamps    






d0f95ba  



Resolves conflicts by integrating traceback timestamp functionality with
the new colorization system and maintaining timestamp regex patterns in tests.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>












" data-pjax="true" class="Link--secondary markdown-title" href="/web/20250601114607/https://github.com/python/cpython/pull/129337/commits/52a1e25d05c09b52eed941e88d4e3c2d590aa026">Fix failing tests when traceback timestamps are enabled    






52a1e25  



Update test_wsgiref, test_pdb, and test_remote_pdb to handle timestamp
suffixes in exception messages when PYTHON_TRACEBACK_TIMESTAMPS is set.

- test_wsgiref: Use traceback.strip_exc_timestamps() before comparing
  error messages in validation tests
- test_pdb: Update doctest expected output to use ellipsis for timestamp
  matching in await support test
- test_remote_pdb: Strip timestamps from stdout before comparison in
  do_test method

All tests now pass both with and without timestamp functionality enabled.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>












" data-pjax="true" class="Link--secondary markdown-title" href="/web/20250601114607/https://github.com/python/cpython/pull/129337/commits/05d6f138a4a921cc2db3108249c96f9343652d11">Add comprehensive tests for traceback timestamps feature    






05d6f13  



Implements the missing tests identified in PR python#129337 TODO section:
- Test pickle/unpickle of all built-in exception types in hierarchy
- Test user-derived exception classes from BaseException, Exception, OSError, ImportError, AttributeError
- Test timestamp presence on all exception types except StopIteration and StopAsyncIteration

Reorganizes tests into a proper package structure with specialized test modules:
- test_basic.py: Original basic functionality tests
- test_pickle.py: Exception pickle/unpickle preservation tests
- test_user_exceptions.py: Custom exception class tests with inheritance validation
- test_timestamp_presence.py: Timestamp behavior verification across all exception types

All tests validate behavior both with and without timestamp feature enabled,
ensuring proper functionality and performance optimizations for StopIteration/StopAsyncIteration.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>












" data-pjax="true" class="Link--secondary markdown-title" href="/web/20250601114607/https://github.com/python/cpython/pull/129337/commits/1fc9c88eb9b5d6936bbcdcabeb5bef5081d9396d">Reduce duplication in traceback timestamps tests    






1fc9c88  



- Create shared_utils.py with common exception creation logic
- Consolidate duplicate code across test_pickle.py, test_timestamp_presence.py, and test_user_exceptions.py
- Embed shared functions directly in subprocess test scripts to avoid import issues
- Reduce overall test code by ~400 lines while preserving functionality
- All 25 tests pass both with and without timestamps enabled

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>












ruff format the tests  






4aaa4dc  











@gpshead gpshead  added  the  🔨 test-with-refleak-buildbots  Test PR w/ refleak buildbots; report in status section  label  May 24, 2025  









Copy link  



bedevere-bot   commented  May 24, 2025  



🤖 New build scheduled with the buildbot fleet by @gpshead for commit 4aaa4dc 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F129337%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.
















@bedevere-bot bedevere-bot  removed  the  🔨 test-with-refleak-buildbots  Test PR w/ refleak buildbots; report in status section  label  May 24, 2025  







gpshead   and others  added 3commits  May 24, 2025 23:41











0 or custom attributes exist - OSError: Apply same optimization while preserving filename attributes - ImportError: Conditionally include state based on meaningful attributes - AttributeError: Always include state dict for Python 3.13 compatibility Results: - ValueError/RuntimeError: 53 bytes (disabled) -> 103 bytes (enabled) - OSError: 56 bytes (disabled) -> 106 bytes (enabled) - AttributeError: 76 bytes (disabled) -> 120 bytes (enabled) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude " data-pjax="true" class="Link--secondary markdown-title" href="/web/20250601114607/https://github.com/python/cpython/pull/129337/commits/ec22c193c2d101191bcc6e1be06f33421467e163">Optimize exception pickle sizes when timestamps are disabled    






ec22c19  



When traceback timestamps are disabled, exception pickles no longer
include an empty state dictionary, reducing pickle size to match
Python 3.13 baseline. When timestamps are enabled, the state dict
is included with timestamp data.

- BaseException: Only include dict when timestamp > 0 or custom attributes exist
- OSError: Apply same optimization while preserving filename attributes
- ImportError: Conditionally include state based on meaningful attributes
- AttributeError: Always include state dict for Python 3.13 compatibility

Results:
- ValueError/RuntimeError: 53 bytes (disabled) -> 103 bytes (enabled)
- OSError: 56 bytes (disabled) -> 106 bytes (enabled)
- AttributeError: 76 bytes (disabled) -> 120 bytes (enabled)

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>












TESTSUBDIRS += our new dir for test_tools  






1b83faf  













shakes fist at trailing spaces from claude  






1ba3ed4  














Sign up for free  to join this conversation on GitHub.  Already have an account?  Sign in to comment  








Reviewers

@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently will be requested when the pull request is marked ready for review ericsnowcurrently is a code owner

@terryjreedy terryjreedy Awaiting requested review from terryjreedy terryjreedy will be requested when the pull request is marked ready for review terryjreedy is a code owner

@gaogaotiantian gaogaotiantian Awaiting requested review from gaogaotiantian gaogaotiantian will be requested when the pull request is marked ready for review gaogaotiantian is a code owner

@brettcannon brettcannon Awaiting requested review from brettcannon brettcannon will be requested when the pull request is marked ready for review brettcannon is a code owner

@ncoghlan ncoghlan Awaiting requested review from ncoghlan ncoghlan will be requested when the pull request is marked ready for review ncoghlan is a code owner

@warsaw warsaw Awaiting requested review from warsaw warsaw will be requested when the pull request is marked ready for review warsaw is a code owner

@vsajip vsajip Awaiting requested review from vsajip vsajip will be requested when the pull request is marked ready for review vsajip is a code owner

@iritkatriel iritkatriel Awaiting requested review from iritkatriel iritkatriel will be requested when the pull request is marked ready for review iritkatriel is a code owner

@FFY00 FFY00 Awaiting requested review from FFY00 FFY00 will be requested when the pull request is marked ready for review FFY00 is a code owner

@erlend-aasland erlend-aasland Awaiting requested review from erlend-aasland erlend-aasland will be requested when the pull request is marked ready for review erlend-aasland is a code owner

@ezio-melotti ezio-melotti Awaiting requested review from ezio-melotti ezio-melotti will be requested when the pull request is marked ready for review ezio-melotti is a code owner

@hugovk hugovk Awaiting requested review from hugovk hugovk will be requested when the pull request is marked ready for review hugovk is a code owner

@AA-Turner AA-Turner Awaiting requested review from AA-Turner AA-Turner will be requested when the pull request is marked ready for review AA-Turner is a code owner



Assignees
No one assigned

Labels  

None yet



Projects
None yet

Milestone
No milestone



Development

Successfully merging this pull request may close these issues.






2 participants  








Add this suggestion to a batch that can be applied as a single commit.  This suggestion is invalid because no changes were made to the code.  Suggestions cannot be applied while the pull request is closed.  Suggestions cannot be applied while viewing a subset of changes.  Only one suggestion per line can be applied in a batch.  Add this suggestion to a batch that can be applied as a single commit.  Applying suggestions on deleted lines is not supported.  You must change the existing code in this line in order to create a valid suggestion.  Outdated suggestions cannot be applied.  This suggestion has been applied or marked resolved.  Suggestions cannot be applied from pending reviews.  Suggestions cannot be applied on multi-line comments.  Suggestions cannot be applied while the pull request is queued to merge.  Suggestion cannot be applied right now. Please check back later.  




Footer



© 2025 GitHub, Inc.  


Terms  

Privacy  

Security  

Status  

Docs  

Contact  






You cant perform that action at this time.