You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Since scraping is a very integrative process it can fail from many causes. Instead of making the user retry himself, it would be good if there was a retry mechanism in the code that would retry x times if the scraping failed.
I will check more in-depth later, but for now, I see two options:
Use Enable/Disable state for importers like we have for exporters.
Add a retry button on the importer status (in the middle of the screen).
I think option 2 is more user-friendly, since the scraping, by definition, is not stable so you need to retry one of the importers in many cases.
But this option is more complicated since then you have more than one button to trigger the process, so you need to create a locking architecture to avoid a parallel process.
Also, using enable/disable checkbox is keeps the way we are working today- manage the config and save it, and use it as is when clicking on Run. Use a retry button require to create a one-time config object and use it to scrape (it is not a bad thing, I just describing why I want the to implement option 1)
The third option we discussed is to identify the events that require us to retry- the timeout events, for example, and automatically retry the importer.
Issue has been marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Activity
Merge pull request #42 from baruchiro/decryptedImporter
baruchiro commentedon Apr 4, 2021
I will check more in-depth later, but for now, I see two options:
I think option 2 is more user-friendly, since the scraping, by definition, is not stable so you need to retry one of the importers in many cases.
But this option is more complicated since then you have more than one button to trigger the process, so you need to create a locking architecture to avoid a parallel process.
Also, using enable/disable checkbox is keeps the way we are working today- manage the config and save it, and use it as is when clicking on Run. Use a retry button require to create a one-time config object and use it to scrape (it is not a bad thing, I just describing why I want the to implement option 1)
baruchiro commentedon Jul 12, 2021
The third option we discussed is to identify the events that require us to retry- the timeout events, for example, and automatically retry the importer.
chore(release): 1.16.0 [skip ci]
github-actions commentedon Nov 10, 2024
Issue has been marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.