Skip to content

c4/multilingual produces Dataflow job file too big (38MB >> 10MB)Β #2711

Closed

Description

@versae

Short description
We are trying to extract the Norwegian (and eventually other Nordic languages) portion of c4/multilingual. Since there is no easy way to download only the data for one language, we are processing the entire c4/multilingual corpus first.

Environment information

  • Operating System: Debian GNU/Linux 10

  • Python version: Python 3.8.5 (miniconda)

  • tensorflow-datasets/tfds-nightly version: 4.1.0 / 4.1.0.dev202011080107

  • tensorflow/tf-nightly version: 2.3.1 / 2.5.0.dev20201108 (tried with and without tf-nightly)

  • Does the issue still exists with the last tfds-nightly package (pip install --upgrade tfds-nightly) ?
    Yes, it does. We read all issues related to C4 and incorporated the necessary changes: pinning dill version and adding options for 450 workers and experiments=shuffle_mode=service in Apache Beam.

Reproduction instructions
On a clean VM with 8vCPU and 32GB of RAM, we installed miniconda and run the next commands:

DATASET_NAME=c4
DATASET_CONFIG=multilingual
GCP_PROJECT=...
GCS_BUCKET=...
GCS_BUCKET_REGION=...

# Add all 72 dumps
rm wet.paths.urls
echo "CC-MAIN-2013-20" >> wet.paths.urls
...
echo "CC-MAIN-2020-40" >> wet.paths.urls

# Put them in the bucket
for wetpath in `cat wet.paths.urls` ; do curl -s https://commoncrawl.s3.amazonaws.com/crawl-data/$wetpath/wet.paths.gz | gunzip | pv --name $wetpath --bytes | gsutil -q cp - "$GCS_BUCKET/tensorflow_datasets/downloads/manual/crawl-data/$wetpath/web.paths" ; done

# Prepare requirements
rm /tmp/beam_requirements.txt
echo "tensorflow_datasets[$DATASET_NAME]" >> /tmp/beam_requirements.txt
echo "tfds-nightly[gcp,$DATASET_NAME]" >> /tmp/beam_requirements.txt
echo "google-apitools" >> /tmp/beam_requirements.txt
# there's an error with avro-python3 and dill, dill version needs to be fixed
# https://github.com/tensorflow/datasets/issues/2636#issuecomment-722551597
echo "dill==0.3.1.1" >> /tmp/beam_requirements.txt
python -m pip install tensorflow tf-nightly
python -m pip install -r /tmp/beam_requirements.txt

# Run main command
python -m tensorflow_datasets.scripts.download_and_prepare \
  --datasets=$DATASET_NAME/$DATASET_CONFIG \
  --data_dir=$GCS_BUCKET/tensorflow_datasets \
  --beam_pipeline_options=\
"region=$GCS_BUCKET_REGION,runner=DataflowRunner,project=$GCP_PROJECT,job_name=$DATASET_NAME-gen,"\
"staging_location=$GCS_BUCKET/binaries,temp_location=$GCS_BUCKET/temp,"\
"dataflow_job_file=gs://$GCS_BUCKET/job_file.json,"\
"requirements_file=/tmp/beam_requirements.txt,max_num_workers=450,experiments=shuffle_mode=service" 2>&1 | tee nb-mc4.log

Link to logs
We removed information about our project and bucket in the logs:

Expected behavior
We would have expected for the script to successfully launch the pipeline in Dataflow, but the JSON job file seems to be too big (37.5MB when max is 10MB), therefore all we get is a Your client issued a request that was too large error message (formatted as a HTML page in the console output).

Sample of the output

  <title>Error 413 (Request Entity Too Large)!!1</title>
  <p><b>413.</b> <ins>ThatοΏ½οΏ½οΏ½s an error.</ins>
  <p>Your client issued a request that was too large.</p>
  <ins>ThatοΏ½οΏ½οΏ½s all we know.</ins>

Additional context
If there is any other way to extract a language portion of c4/multilingual we'd be eager to try it as well.

Update (March 1st, 2021): Instructions to successfully run the pipeline using one dump are detailed in the #2711 (comment).

Activity

  1. Conchylicultor commented on Nov 9, 2020

    @Conchylicultor
    Contributor
  2. adarob commented on Nov 9, 2020

    @adarob
    Member

    @versae I'd recommend generating all languages you want at the same time since it has to process all of the data either way. However, I think if you added a new config with only the languages you want, you'd avoid the JSON issue.

    @Conchylicultor we would need to allow them to add a new config to the dataset with only the languages they are interested in. What's the recommended way for them to do that?

  3. versae commented on Nov 9, 2020

    @versae
    Author

    Thanks for the quick replies, @Conchylicultor and @adarob. We thought about the custom config, but had troubles figuring out how to make it work. Even with a custom config, it's still not clear to us how to run the download_and_prepare code, either in command or script mode.

    After inspecting the source code, it seems it should be possible to only process the data for one language, even if we have to download everything, but not sure about how to do it.

  4. adarob commented on Nov 9, 2020

    @adarob
    Member

    You should be able to add a config similar to 'multilingual', but with only the nordic languages listed. Let's say you name it
    'nordic'. Then you can call download_and_prepare with c4/nordic as the dataset.

    You'd make the change to a local clone of the repo and then you can run pip install -e . to install it on the master VM.

  5. versae commented on Nov 9, 2020

    @versae
    Author

    I see. It feels unnecessarily complicated, but we'll give it a try 🀞 Really, it'd be great if some jobfile size control was allowed or implemented directly within the download_and_prepare script. The official docs recommend to restructure the pipeline to avoid such errors. We hope downloading only Nordic or Norwegian languages would do the trick, but honestly it seems the issue is a bit deeper.

    I wonder, if the limit it's 10MB and the current code generates a jobfile of almost 40MB, is there a way to lift, even temporarily, this limitation to at least be able to launch the job?

    In any case, thanks for the help.

  6. adarob commented on Nov 9, 2020

    @adarob
    Member

    The issue is the large number of splits (>100) that are produced by this pipeline. Limiting to ~10 languages should reduce the json size by a factor of 10, I believe.

    Other options to fix the deeper problem would involve either having the DataFlow team raise the limit or merge some of the downstream steps in the TFDS sharding and writing portions of the pipeline.

  7. versae commented on Nov 9, 2020

    @versae
    Author

    Thinking about, if the downloading and processing of the data happens in the workers, the information about the languages is not available until the query is already sent and in execution, isn't it? Limiting the languages, or for that matter, selecting specific splits, would still generate the same jobfile to be sent to Dataflow I believe.

  8. adarob commented on Nov 9, 2020

    @adarob
    Member

    The actual pipeline "blueprint" is pre-generated and sent to the DataFlow service, which is where I think your issue is. This blueprint includes all of the per-language stages.

  9. versae commented on Nov 9, 2020

    @versae
    Author

    We finally tried our custom config for Norwegian (excluding no-validation) but the jobfile is still too big (10.2MB), so Dataflow is rejecting the request for 200KB :\

  10. adarob commented on Nov 9, 2020

    @adarob
    Member

    Can you try also removing c4_utils.UNKNOWN_LANGUAGE here

    for lang in self.builder_config.languages + [c4_utils.UNKNOWN_LANGUAGE]:

  11. versae commented on Nov 9, 2020

    @versae
    Author

    Tried with and without that line. The request now went through but Dataflow still failed and complained about size, since it's still a tiny bit larger than 10MB (~10.2MB)

    {
      "error": {
        "code": 400,
        "message": "(ffddc5dbfa10dae9): The job graph is too large. Please try again with a smaller job graph, or split your job into two or more smaller jobs.",
        "status": "INVALID_ARGUMENT"
      }
    }

    Adding the updated job_file.json (compressed) in case it's useful. There are a couple of big chunks of binary data taking most of the size of the file.

    Wondering if the hard limit of 10MB could be lifted temporarily at least or for individual projects.

  12. adarob commented on Nov 10, 2020

    @adarob
    Member

    I'll see if we can get some help from the dataflow team.

  13. versae commented on Nov 10, 2020

    @versae
    Author

    Thank you so much.

  14. rezarokni commented on Nov 10, 2020

    @rezarokni

    If possible its best to reduce the nodes in the graph, for example multiplexing the values into a source. @adarob we should look at the pipeline in more detail to look at that.

    In the mean time you can make use of --experiments=upload_graph with the Dataflow pipeline arguments, which allow larger than 10MB pipelines. Although note things like the UI will have limitations with this experimental flag.

  15. 89 remaining items

  16. versae commented on Dec 21, 2020

    @versae
    Author

    Sure, the languages we're interested in are Norwegian, Swedish, Danish, Icelandic, an Faroese (although I think this one is not included in mC4). The ISO codes are no, sv, da, and is (Faroese would be fo).

  17. versae commented on Dec 21, 2020

    @versae
    Author

    It'd be also great to know how many words or GB of raw text would be.

  18. adarob commented on Dec 29, 2020

    @adarob
    Member

    First off, I found a significant bottleneck that reduces the parallelism to ~71 instead of your number of workers. It will be fixed in #2895.

    Here are the number of documents from 1 common crawl dump:

    beam:ParDo(_PredictLanguageFn):MetricName(namespace=language-filter, name=passed:da) | 3,163,698 (15 GB)
    beam:ParDo(_PredictLanguageFn):MetricName(namespace=language-filter, name=passed:is) | 219,188 (1 GB)
    beam:ParDo(_PredictLanguageFn):MetricName(namespace=language-filter, name=passed:no) | 2,617,118 (12 GB)
    beam:ParDo(_PredictLanguageFn):MetricName(namespace=language-filter, name=passed:sv) | 5,900,912 (26 GB)
    
  19. acul3 commented on Jan 8, 2021

    @acul3

    hi @versae thanks for creating the issue...
    did you guys finish it successfully(extracting the specific language from mc4) ?
    if it did..how long does it take?

    i'm also planning to extract specific language from mc4 using dataflow beam dataset..
    any tips to prepare it or maybe code how to do it would be helpful
    thanks

  20. versae commented on Feb 9, 2021

    @versae
    Author

    Hi!

    Sorry for the long hiatus. Thanks, @adarob. We're finally resuming work on this and I hope to start the processing in the coming days. We might just start with the first, last, and in between dumps just to get an idea of how long and how big. But the estimates in number of documents per language really help.

    @acul3, we were not able to finish the processing even after starting 500 workers for over 36 hours. We'll prepare everything again and report back. If you have any insights I'd be eager to know as well.

    Cheers.

  21. versae commented on Feb 25, 2021

    @versae
    Author

    I was finally able to try this again. A few things have changed and it seems the experimental flags are not needed anymore. The setup is as follows:

    python -m tensorflow_datasets.scripts.download_and_prepare \
      --datasets=$DATASET_NAME/$DATASET_CONFIG \
      --data_dir=$GCS_BUCKET/tensorflow_datasets \
      --beam_pipeline_options="region=$GCS_BUCKET_REGION,runner=DataflowRunner,project=$GCP_PROJECT,job_name=$DATASET_NAME-$DATASET
    _CONFIG-1dump-gen,staging_location=$GCS_BUCKET/binaries,temp_location=$GCS_BUCKET/temp,dataflow_job_file=$GCS_BUCKET/job_file.json,requirements_file=/
    tmp/beam_requirements.txt,autoscaling_algorithm=NONE,num_workers=50" 2>&1 | tee nb-mc4-1dump.log

    The beam_requirements.txt looks as follows:

    apache-beam[gcp]
    tensorflow-datasets[c4]
    google-apitools
    dill<0.3.2,>=0.3.1.1
    

    The last main process logs:

    INFO[dataflow_runner.py]: 2021-02-24T10:48:08.070Z: JOB_MESSAGE_DEBUG: Value "no-validation_write/GroupBucketsAndBoundaries/GroupByKey/Session" materialized.
    INFO[dataflow_runner.py]: 2021-02-24T10:48:08.095Z: JOB_MESSAGE_DEBUG: Value "no-validation_write/GroupShards/Session" materialized.                 
    INFO[dataflow_runner.py]: 2021-02-24T10:48:08.132Z: JOB_MESSAGE_DEBUG: Value "sv_write/GroupByBucket/Session" materialized.                          
    INFO[dataflow_runner.py]: 2021-02-24T10:48:08.160Z: JOB_MESSAGE_DEBUG: Value "sv_write/GroupBucketsAndBoundaries/GroupByKey/Session" materialized.   
    INFO[dataflow_runner.py]: 2021-02-24T10:48:08.196Z: JOB_MESSAGE_DEBUG: Value "sv_write/CombineBucketsSizes/CombineBucketsSizes/CombinePerKey/GroupByKey/Session" materialized.
    

    I run it for 24 hours only on one dump, CC-MAIN-2013-20. Everything run fine for approximately 11 hours, as I could see data being written to the bucket, counters counting, and workers CPU working at 90%-95% capacity. Then, it all suddenly stop. Workers CPU usage dropped to 4%, counters stopped, and the bucket did not grow anymore. No critical nor regular errors in the logs. No budget problem. Nothing. I really have no clue what is going on. I just restarted the process to see if anything changes.

    image

    CPU utilization (All Workers)

    I'd be happy to open a new issue if that's more appropriate.

  22. Conchylicultor commented on Feb 25, 2021

    @Conchylicultor
    Contributor

    If the input pipeline entered a phase where it is IO bound, it might be normal that the CPU rate drop.

  23. versae commented on Feb 25, 2021

    @versae
    Author

    The throughput also drops to zero for all the pipeline stages, and although progress is 100% is not marked as "succeeded". For example:

    image

  24. versae commented on Feb 26, 2021

    @versae
    Author

    It seems like I got past that problem on a second run, but it has now failed with a "No space left on the device" error that I guess is coming from the workers as they try to write too many or too big temporary files.

    File "/usr/local/lib/python3.8/site-packages/dataflow_worker/shuffle.py", line 282, in __next__ return next(self.iterator)
    File "/usr/local/lib/python3.8/site-packages/dataflow_worker/shuffle.py", line 240, in __iter__ chunk, next_position = self.reader.Read(start_position, end_position)
    File "third_party/windmill/shuffle/python/shuffle_client.pyx", line 135, in shuffle_client.PyShuffleReader.Read OSError: Shuffle read failed: b'INTERNAL: RPC error: IO error: /var/shuffle/sorted-dataset-1/1818: No space left on device, {"created":"@1614294826.207181169","description":"Error received from peer ipv4:10.128.0.123:12346","file":"third_party/grpc/src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"IO error: /var/shuffle/sorted-dataset-1/1818: No space left on device","grpc_status":13} when c4-nordic-1dump-gen-02250406-zsv5-harness-s9tq talking to c4-nordic-1dump-gen-02250406-zsv5-harness-rrsn:12346. [type.googleapis.com/util.MessageSetPayload=\'[dist_proc.dax.internal.TrailProto] { trail_point { source_file_loc { filepath: "dist_proc/dax/shuffle/sorter/shuffle_client_key_value_iterator.cc" line: 84 } } trail_point { source_file_loc { filepath: "dist_proc/dax/shuffle/sorter/chunked_sorted_reader.cc" line: 73 } } }\']'
    
            at __iter__ (/usr/local/lib/python3.8/site-packages/dataflow_worker/shuffle.py:441)
            at dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start (shuffle_operations.py:82)
            at dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start (shuffle_operations.py:80)
            at dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start (shuffle_operations.py:79)
            at dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start (shuffle_operations.py:64)
            at dataflow_worker.shuffle_operations.GroupedShuffleReadOperation.start (shuffle_operations.py:63)
            at execute (/usr/local/lib/python3.8/site-packages/dataflow_worker/executor.py:179)
            at do_work (/usr/local/lib/python3.8/site-packages/dataflow_worker/batchworker.py:649)
    

    Not sure what's a proper disk size for the workers, I'm using whatever is the default now. I also don't know if I should run it again with experiments=shuffle_mode=service or if that's just irrelevant at this point.

  25. adarob commented on Feb 26, 2021

    @adarob
    Member

    experiments=shuffle_mode=service should reduce the files, but potentially only on GCS. @daphnei has also recently seen this issue. It appears to be new.

  26. tvalentyn commented on Feb 26, 2021

    @tvalentyn

    If the temporary files are created by shuffle, --experiments=shuffle_mode=service should help;
    I think the default --disk_size_gb disk size is 250G; with shuffle service it is 25G.

  27. versae commented on Mar 1, 2021

    @versae
    Author

    It finished! πŸŽ‰ It successfully processed the 105TB and almost 2 billion files of the CC-MAIN-2013-20 dump. Using 75 workers, it took 31 hours. That's roughly 3.3TB per hour. I increased disk size to 100GB (it might be unnecessarily big), force v2 of the runner, and enabled shuffle mode. The total cost was around 950€.

    DATASET_NAME=c4
    DATASET_CONFIG=nordic
    GCP_PROJECT=...
    GCS_BUCKET=...
    GCS_BUCKET_REGION=...

    The command looks like this:

    python -m tensorflow_datasets.scripts.download_and_prepare \
      --datasets=$DATASET_NAME/$DATASET_CONFIG \
      --data_dir=$GCS_BUCKET/tensorflow_datasets \
      --beam_pipeline_options="region=$GCS_BUCKET_REGION,runner=DataflowRunner,project=$GCP_PROJECT,job_name=$DATASET_NAME-$DATASET_CONFIG-1dump-gen,staging_location=$GCS_BUCKET/binaries,temp_location=$GCS_BUCKET/temp,dataflow_job_file=$GCS_BUCKET/job_file.json,requirements_file=/tmp/beam_requirements.txt,autoscaling_algorithm=NONE,disk_size_gb=100,num_workers=75,experiments=shuffle_mode=service,experiments=use_runner_v2," 2>&1 | tee nb-mc4-1dump.log

    The beam_requirements.txt file contains:

    apache-beam[gcp]
    tensorflow-datasets[c4]
    google-apitools
    dill<0.3.2,>=0.3.1.1
    

    And I patched locally 0db70eb to include a nordic config and ignore the unknown languages: nordic_dump2013-20.patch.txt.

    I did not delete the contents of the bucket from the previous run. Do runs cache any part of the process so after a failed run it takes shorter to run?

  28. daphnei commented on Mar 9, 2021

    @daphnei
    Contributor

    Congrats on getting this working!

    Out of curiosity, did you have to request a lot of quota increases from GCS to make it work, and if so what quotas did you get increases to?

    The last time I tried this, I kept running into various out-of-quota warnings.

  29. sumanthd17 commented on Mar 9, 2021

    @sumanthd17

    @versae Congratulations on getting this work.

    I would like to understand the finer details of this work. We are planning on doing this for english. Is it possible to set up a meeting to discuss this.

    Thanks in Advance

  30. versae commented on Mar 9, 2021

    @versae
    Author

    Thanks, @daphnei, @sumanthd17.

    @daphnei, the quota thing was the first problem I had. Thankfully, it was fixed by my manager. I'm my experience, having a budget estimate on resources and cost really helps when justifying how much money you need to run experiments.

    @sumanthd17, I am not affiliated with Google. I'm sure that @adarob and @rezarokni know way better the internals of their own work. That being said, happy to help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions