Skip to content

Add a retry mechanism for scraping failureΒ #42

Closed as not planned
Closed as not planned

Description

@brafdlog

Since scraping is a very integrative process it can fail from many causes. Instead of making the user retry himself, it would be good if there was a retry mechanism in the code that would retry x times if the scraping failed.

Activity

  1. added a commit that references this issue on Jun 5, 2020
  2. baruchiro commented on Apr 4, 2021

    @baruchiro
    Collaborator

    I will check more in-depth later, but for now, I see two options:

    1. Use Enable/Disable state for importers like we have for exporters.
    2. Add a retry button on the importer status (in the middle of the screen).

    I think option 2 is more user-friendly, since the scraping, by definition, is not stable so you need to retry one of the importers in many cases.

    But this option is more complicated since then you have more than one button to trigger the process, so you need to create a locking architecture to avoid a parallel process.
    Also, using enable/disable checkbox is keeps the way we are working today- manage the config and save it, and use it as is when clicking on Run. Use a retry button require to create a one-time config object and use it to scrape (it is not a bad thing, I just describing why I want the to implement option 1)

  3. baruchiro commented on Jul 12, 2021

    @baruchiro
    Collaborator

    The third option we discussed is to identify the events that require us to retry- the timeout events, for example, and automatically retry the importer.

  4. added a commit that references this issue on Dec 4, 2023
  5. github-actions commented on Nov 10, 2024

    @github-actions
    Contributor

    Issue has been marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions