GitHub Actions Failures #1408

Activity

1. ryanjbaxter

   added

   bug

   on Aug 10, 2023
2. wind57 commented on Aug 10, 2023

   

   wind57

   on Aug 10, 2023

   Last edited by wind57

   Contributor

   More actions

   The previous step needs to download some artifacts, the timeout to do that is 30 minutes... The next step fails to find it, of course. Either way, that is very annoying indeed :( I'll take a closer look.

   👍React with 👍1Reacted by Ryan Baxter
3. wind57 commented on Aug 15, 2023

   

   wind57

   on Aug 15, 2023

   Contributor

   More actions

   a few things for history here, as I have started working on it. The process is this atm:

   • build docker images needed for our integration tests
   • save these images (via actions/cache@v3)
   • start matrix steps to run the tests (we do this in parallel)
   • in each step of the matrix, download those docker images (via same actions/cache@v3)

   Of course, the same exact images are downloaded for each matrix step. One might think that this is at least sub-optimal, and we could download them once, per all matrix steps. Something like this in pseudo-code:

     test_when_cache_present:
       needs: [ build ]
       runs-on: ubuntu-latest
       env:
         SEGMENT_DOWNLOAD_TIMEOUT_MINS: 30
       # only run this one if there is a previous cache of test times
       if: needs.build.outputs.test_times_cache_present == 'true'
       timeout-minutes: 60
   
       #####################################################
       # have some sort of a "step" here that would download docker images
       #####################################################
   
       strategy:
         fail-fast: true
         matrix:
           current_index: [ "${{ fromJSON(needs.build.outputs.matrix_array) }}" ]
           number_of_jobs: [ "${{ fromJSON(needs.build.outputs.number_of_matrix_instances) }}" ]
   

   Unfortunately, github actions does not support such a thing, and this is why we download the same images in every single matrix step. If at least one step timeouts (which happens in our case), the entire flow breaks and this is exactly what we see.
4. ryanjbaxter commented on Aug 15, 2023

   

   ryanjbaxter

   on Aug 15, 2023

   ContributorAuthor

   More actions

   What does it mean for a step to timeout?
5. wind57 commented on Aug 15, 2023

   

   wind57

   on Aug 15, 2023

   Last edited by wind57

   Contributor

   More actions

   this

   Basically download of a cache is "stuck", as seen in our logs:

   Received 3351248896 of 3790969212 (88.4%), 24.0 MBs/sec
   Received 3351248896 of 3790969212 (88.4%), 23.8 MBs/sec
   Received 3351248896 of 3790969212 (88.4%), 23.6 MBs/sec
   Received 3351248896 of 3790969212 (88.4%), 23.5 MBs/sec
   Received 3351248896 of 3790969212 (88.4%), 23.3 MBs/sec
   Received 3351248896 of 3790969212 (88.4%), 23.1 MBs/sec
   Received 3351248896 of 3790969212 (88.4%), 23.0 MBs/sec
   Received 3351248896 of 3790969212 (88.4%), 22.8 MBs/sec
   Received 3351248896 of 3790969212 (88.4%), 22.6 MBs/sec
   Received 3351248896 of 3790969212 (88.4%), 22.5 MBs/sec
   Received 3351248896 of 3790969212 (88.4%), 22.3 MBs/sec
   Received 3351248896 of 3790969212 (88.4%), 22.2 MBs/sec
   Received 3351248896 of 3790969212 (88.4%), 21.9 MBs/sec
   Received 3351248896 of 3790969212 (88.4%), 21.7 MBs/sec
   Received 3351248896 of 3790969212 (88.4%), 21.6 MBs/sec
   Received 3351248896 of 3790969212 (88.4%), 21.4 MBs/sec
   Received 3351248896 of 3790969212 (88.4%), 21.4 MBs/sec
   Warning: Failed to restore: The operation cannot be completed in timeout.
   Cache not found for input keys: docker-images-cache-5824481925, docker-images-cache-5824481925
   

   👍React with 👍1Reacted by Ryan Baxter
6. ryanjbaxter

   linked a pull request that will close this issuereplace github cache with buildjet cache #1409on Aug 15, 2023
7. wind57 commented on Aug 19, 2023

   

   wind57

   on Aug 19, 2023

   Contributor

   More actions

   we can close this one btw
8. ryanjbaxter

   added this to 2021.0.9and 2022.0.5on Aug 19, 2023
9. ryanjbaxter

   modified the milestones: 2.1.6, 3.0.5 on Aug 19, 2023
10. ryanjbaxter

    closed this as completedon Aug 19, 2023
11. github-project-automation

    moved this to Done in 2022.0.5on Aug 19, 2023
12. github-project-automation

    moved this to Done in 2021.0.9on Aug 19, 2023