Skip to content

NamedTemporaryFile could cause double-close on an fd if _TemporaryFileWrapper throws #83499

Closed

Description

@nneonneo
mannequin
BPO 39318
Nosy @nedbat, @sethtroisi, @albertz, @serhiy-storchaka, @eryksun, @jstasiak, @sanjioh, @joernheissler
PRs
  • bpo-39318: Catch a failure in NamedTemporaryFile to prevent leaking a descriptor #17985
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/serhiy-storchaka'
    closed_at = None
    created_at = <Date 2020-01-13.04:56:47.144>
    labels = ['3.8', 'type-bug', 'library', '3.9', '3.10']
    title = 'NamedTemporaryFile could cause double-close on an fd if _TemporaryFileWrapper throws'
    updated_at = <Date 2021-03-28.03:37:24.913>
    user = 'https://bugs.python.org/nneonneo'

    bugs.python.org fields:

    activity = <Date 2021-03-28.03:37:24.913>
    actor = 'eryksun'
    assignee = 'serhiy.storchaka'
    closed = False
    closed_date = None
    closer = None
    components = ['Library (Lib)']
    creation = <Date 2020-01-13.04:56:47.144>
    creator = 'nneonneo'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 39318
    keywords = ['patch']
    message_count = 10.0
    messages = ['359888', '359892', '359935', '359957', '359959', '360040', '360041', '360071', '360084', '360104']
    nosy_count = 12.0
    nosy_names = ['nneonneo', 'nedbat', 'Seth.Troisi', 'Albert.Zeyer', 'serhiy.storchaka', 'paul_ollis', 'eryksun', 'jstasiak', 'sanjioh', 'joernheissler', 'hitbox', 'rekcartgubnohtyp']
    pr_nums = ['17985']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue39318'
    versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

    Activity

    1. nneonneo commented on Jan 13, 2020

      nneonneomannequin
      MannequinAuthor

      tempfile.NamedTemporaryFile creates its wrapper like so:

      try:
          file = _io.open(fd, mode, buffering=buffering,
                          newline=newline, encoding=encoding, errors=errors)
      
              return _TemporaryFileWrapper(file, name, delete)
          except BaseException:
              _os.unlink(name)
              _os.close(fd)
              raise

      If _TemporaryFileWrapper throws any kind of exception (even KeyboardInterrupt), this closes fd but leaks a valid file pointing to that fd. The file will later attempt to close the fd when it is collected, which can lead to subtle bugs. (This particular issue contributed to this bug: https://nedbatchelder.com/blog/202001/bug_915_please_help.html)

      This should probably be rewritten as:

      try:
          file = _io.open(fd, mode, buffering=buffering,
                          newline=newline, encoding=encoding, errors=errors)
      except:
          _os.unlink(name)
          _os.close(fd)
          raise
      
      try:
          return _TemporaryFileWrapper(file, name, delete)
      except BaseException:
          _os.unlink(name)
          file.close()
          raise
      

      or perhaps use nested try blocks to avoid the _os.unlink duplication.

    2. added
      stdlibStandard Library Python modules in the Lib/ directory
      type-bugAn unexpected behavior, bug, or error
      on Jan 13, 2020
    3. albertz commented on Jan 13, 2020

      albertzmannequin
      Mannequin

      Instead of `except:` and `except BaseException:`, I think better use `except Exception:`.

      For further discussion and reference, also see the discussion here: https://news.ycombinator.com/item?id=22028581

    4. serhiy-storchaka commented on Jan 13, 2020

      @serhiy-storchaka
      Member

      This issue is more complex. I am working on it.

    5. 10 remaining items

    6. transferred this issue fromon Apr 10, 2022
    7. rouilj commented on Jun 15, 2022

      @rouilj

      Just a heads up in python 3.11 on Travis CI I am seeing a similar issue with sqlite throwing an I/O error
      randomly during testing. Some tests always fail, other tests randomly fail. E.g.

      https://app.travis-ci.com/github/roundup-tracker/roundup/jobs/573649310#L5779
      for a sample failure. Note that the same suite works fine with 3.8 and 3.10.

      So 3.11 might be more prone to this. Does it make sense to add 3.11 to the labels on this ticket.

    8. added a commit that references this issue on Jun 15, 2022
    9. serhiy-storchaka commented on Jun 17, 2022

      @serhiy-storchaka
      Member

      See #93874. It was mostly ready (except a NEWS entry and some polishing) more than 2 years ago. Now I do not remember why it was not finished at that time -- either I just put it off until the next morning, but was distracted by other issues and forgot about it, or I found some issues with it, or I found a better way. So please make a careful review.

      Maybe it was deferred because there is a very tiny possibility of a file descriptor leak if you interrupt the program between os.open() and return. It is a part of more general issue and is not solvable at the moment.

    10. added a commit that references this issue on Jun 26, 2022
    11. added a commit that references this issue on Jun 26, 2022
    12. added a commit that references this issue on Jun 26, 2022
    13. added 3 commits that reference this issue on Jun 26, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Labels

    3.10 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions