Skip to content

test_cprofile crashed on main ย #95045

Closed
Closed

Description

@kumaraditya303

Commit tested: 1834133

Reproducer:

./python -m test
Modules/gcmodule.c:442: update_refs: Assertion "gc_get_refs(gc) != 0" failed
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7fe79c68cbb0
object refcount : 0
object type     : 0x55d1aa0d6f10
object type name: _lsprof.Profiler
object repr     : <refcnt 0 at 0x7fe79c68cbb0>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007fe7a2694280 (most recent call first):
  Garbage-collecting
  File "/workspaces/cpython/Lib/test/test_cprofile.py", line 29 in test_bad_counter_during_dealloc
  File "/workspaces/cpython/Lib/unittest/case.py", line 579 in _callTestMethod
  File "/workspaces/cpython/Lib/unittest/case.py", line 623 in run
  File "/workspaces/cpython/Lib/unittest/case.py", line 678 in __call__
  File "/workspaces/cpython/Lib/unittest/suite.py", line 122 in run
  File "/workspaces/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/workspaces/cpython/Lib/unittest/suite.py", line 122 in run
  File "/workspaces/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/workspaces/cpython/Lib/unittest/suite.py", line 122 in run
  File "/workspaces/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/workspaces/cpython/Lib/test/support/testresult.py", line 140 in run
  File "/workspaces/cpython/Lib/test/support/__init__.py", line 1090 in _run_suite
  File "/workspaces/cpython/Lib/test/support/__init__.py", line 1216 in run_unittest
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 281 in _test_module
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 317 in _runtest_inner2
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 360 in _runtest_inner
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 235 in _runtest
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 265 in runtest
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 450 in run_tests_sequential
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 564 in run_tests
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 742 in _main
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 701 in main
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 763 in main
  File "/workspaces/cpython/Lib/test/__main__.py", line 2 in <module>
  File "/workspaces/cpython/Lib/runpy.py", line 88 in _run_code
  File "/workspaces/cpython/Lib/runpy.py", line 198 in _run_module_as_main

Extension modules: _testcapi, _xxsubinterpreters, _testinternalcapi, _testbuffer, _testmultiphase (total: 5)
Aborted (core dumped)

Interestingly it does not crashes if run with multiple processes.

cc @pablogsal

Activity

  1. added
    type-crashA hard crash of the interpreter, possibly with a core dump
    3.12only security fixes
    on Jul 20, 2022
  2. pablogsal commented on Jul 20, 2022

    @pablogsal
    Member

    Does this happen on 3.11?

  3. kumaraditya303 commented on Jul 20, 2022

    @kumaraditya303
    ContributorAuthor

    3.11 does not seem affected, I tried running tests but haven't got any crashes yet.

  4. pablogsal commented on Jul 23, 2022

    @pablogsal
    Member

    I am not able to reproduce this on my laptop. Will try again on my Linux machine later today

  5. kumaraditya303 commented on Jul 24, 2022

    @kumaraditya303
    ContributorAuthor

    I am able to reproduce on Linux with the current HEAD main.

  6. pablogsal commented on Jul 24, 2022

    @pablogsal
    Member

    Can you bisect it?

  7. pablogsal commented on Jul 24, 2022

    @pablogsal
    Member

    Cannot reproduce in my Linux box:

    ./python -m test
    == CPython 3.12.0a0 (heads/main:23f6944c37, Jul 24 2022, 15:14:06) [GCC 12.1.0]
    == Linux-5.18.13-arch1-1-x86_64-with-glibc2.35 little-endian
    == cwd: /home/pablogsal/github/python/main/build/test_python_2216269รฆ
    == CPU count: 36
    == encodings: locale=UTF-8, FS=utf-8
    
     ........
    
    == Tests result: SUCCESS ==
    
    417 tests OK.
    
    19 tests skipped:
        test_curses test_devpoll test_kqueue test_launcher test_msilib
        test_ossaudiodev test_smtpnet test_socketserver test_startfile
        test_tix test_tkinter test_ttk test_urllib2net test_urllibnet
        test_winconsoleio test_winreg test_winsound test_xmlrpc_net
        test_zipfile64
    
    Total duration: 40 min 15 sec
    Tests result: SUCCESS
    
  8. kumaraditya303 commented on Jul 27, 2022

    @kumaraditya303
    ContributorAuthor

    I can reproduce on main HEAD 78eb3f7

    Crash:

    Modules/gcmodule.c:442: update_refs: Assertion "gc_get_refs(gc) != 0" failed
    Enable tracemalloc to get the memory block allocation traceback
    
    object address  : 0x7ff199c0b8c0
    object refcount : 0
    object type     : 0x556cad2891e0
    object type name: _lsprof.Profiler
    object repr     : <refcnt 0 at 0x7ff199c0b8c0>
    
    Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
    Python runtime state: initialized
    
    Current thread 0x00007ff19d4e6280 (most recent call first):
      Garbage-collecting
      File "/workspaces/cpython/Lib/test/test_cprofile.py", line 29 in test_bad_counter_during_dealloc
      File "/workspaces/cpython/Lib/unittest/case.py", line 579 in _callTestMethod
      File "/workspaces/cpython/Lib/unittest/case.py", line 623 in run
      File "/workspaces/cpython/Lib/unittest/case.py", line 678 in __call__
      File "/workspaces/cpython/Lib/unittest/suite.py", line 122 in run
      File "/workspaces/cpython/Lib/unittest/suite.py", line 84 in __call__
      File "/workspaces/cpython/Lib/unittest/suite.py", line 122 in run
      File "/workspaces/cpython/Lib/unittest/suite.py", line 84 in __call__
      File "/workspaces/cpython/Lib/unittest/suite.py", line 122 in run
      File "/workspaces/cpython/Lib/unittest/suite.py", line 84 in __call__
      File "/workspaces/cpython/Lib/test/support/testresult.py", line 140 in run
      File "/workspaces/cpython/Lib/test/support/__init__.py", line 1090 in _run_suite
      File "/workspaces/cpython/Lib/test/support/__init__.py", line 1216 in run_unittest
      File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 281 in _test_module
      File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 317 in _runtest_inner2
      File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 360 in _runtest_inner
      File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 235 in _runtest
      File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 265 in runtest
      File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 450 in run_tests_sequential
      File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 564 in run_tests
      File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 742 in _main
      File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 701 in main
      File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 763 in main
      File "/workspaces/cpython/Lib/test/__main__.py", line 2 in <module>
      File "/workspaces/cpython/Lib/runpy.py", line 88 in _run_code
      File "/workspaces/cpython/Lib/runpy.py", line 198 in _run_module_as_main
    
    Extension modules: _testcapi, _xxsubinterpreters, _testinternalcapi, _testbuffer, _testmultiphase (total: 5)
    Aborted (core dumped)
    

    Environment:

    == CPython 3.12.0a0 (heads/main:78eb3f79ea, Jul 27 2022, 10:43:00) [GCC 9.4.0]
    == Linux-5.4.0-1074-azure-x86_64-with-glibc2.31 little-endian
    == cwd: /workspaces/cpython/build/test_python_4447รฆ
    == CPU count: 4
    == encodings: locale=UTF-8, FS=utf-8

    Another things is that if you remove test_asyncio then it passes, my guess is that it has something to do with the code specialization triggered by test_asyncio as that test executes a lot of code.

    cc @markshannon @brandtbucher

    I'll bisect later.

  9. kumaraditya303 commented on Jul 27, 2022

    @kumaraditya303
    ContributorAuthor

    Turns out the profiler's deallocator did not untracked the object leading to the gc crash.

    See #95315

  10. added 5 commits that reference this issue on Jul 27, 2022
  11. kumaraditya303 commented on Jul 27, 2022

    @kumaraditya303
    ContributorAuthor

    Fixed by #95315

  12. Repository owner moved this from Todo to Done in Release and Deferred blockers ๐Ÿšซon Jul 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

3.10 (EOL)end of life3.11only security fixes3.12only security fixestype-crashA hard crash of the interpreter, possibly with a core dump

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions