Skip to content

Multi-threaded media export - #1617

Merged
TheJJ merged 18 commits into
SFTtech:masterfrom
heinezen:feature/export_threaded
Jan 28, 2024
Merged

TheJJ merged 18 commits into
SFTtech:masterfrom
heinezen:feature/export_threaded

Conversation

@heinezen

Copy link
Copy Markdown
Member

Makes large parts of the media export process multi-threaded.

We previously had a multi-threaded solution in the converter that was deactivated when the new converter was. This implementation doesn't reintroduce this behavior and instead opts for a simpler solution using Python's multiprocessing module with Pool, Manager and Queue. Low-level routines were not needed and the solution now works (with adjustments) for all media files.

The method used is based on the main thread loading the file data and feeding it as a producer to the worker threads. The worker threads convert the data and write the result to file.

The speedup is mostly negligable for the earlier game versions (until SWGB) where the files are small and fast to convert, so the main process cannot provide data fast enough to fully use all available resources. The time gains there are mostly eaten up by the overhead of the thread pool. However, there are signifiant speedups for DE1 and DE2 where export times are cut down to 25% of the previous runtime. Here are some before/after export times for comparison:

  • AoE1 (1997)

    • before: 28.410s
    • after: 33.39s
  • AoE2 (1999)

    • before: 81.186s
    • after: 51.01s
  • SWGB

    • before: 109.620s
    • after: 62.07s
  • HD Edition

    • before: 67.717s
    • after: 53.23s
  • DE1

    • before: 216.225s
    • after: 66.48s
  • DE2

    • before: 959.706s
    • after: 250.63s

CPU usage

@heinezen heinezen added improvement Enhancement of an existing component lang: python Done in Python code area: assets Involved with assets (images, sounds, ...) labels Jan 20, 2024
@heinezen
heinezen force-pushed the feature/export_threaded branch from d41e544 to 33b25b9 Compare January 21, 2024 06:10
@heinezen
heinezen force-pushed the feature/export_threaded branch from fe334da to 97a67c7 Compare January 25, 2024 18:37
@TheJJ
TheJJ merged commit 460ea98 into SFTtech:master Jan 28, 2024
@heinezen
heinezen deleted the feature/export_threaded branch February 23, 2024 22:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: assets Involved with assets (images, sounds, ...) improvement Enhancement of an existing component lang: python Done in Python code

Projects

Status: βœ… Done

Development

Successfully merging this pull request may close these issues.

2 participants