Skip to content

Clarify maximum block size #3104

Closed
Closed

Description

@robcat

The current implementation specifies a fixed maximum block size of 1MiB.

This is a pretty fundamental limitation, it should be explained better why this is the case (IPLD does not seem to be limited in the same way).

Motivation

When ipfs/specs#130 will land, it will be straightforward to automatically convert many hashes already used for integrity checks.

For example, Canonical could construct the "ubuntu releases" object (it would contain links to the historical live cd ISOs) and publish it in IPNS. It will just need to convert these hashes in the CIDv1 format, using the "raw data" codec.

This object would have a perfectly legitimate purpose (i.e. proving the integrity of the live cds), but ipfs will not be able to handle its leaves.

Activity

  1. jbenet commented on Aug 21, 2016

    @jbenet
    Member

    Reasons for limitation:

    • security - easy to DOS nodes without forcing small chunks
    • deduplication - small chunks can dedup. big ones effectively dont.
    • latency - can externalize small pieces already (think a stream)
    • bandwidth - optimize the use of bandwidth across many peers
    • performance - much better perf to hold small pieces in memory. hash along the dag to verify the whole thing.

    the big DOS problem with huge leaves is that malicious nodes can serve bogus stuff for a long time before a node can detect the problem (imagine having to download 4GB before you can check whether any of it is valid). this was super harmful for bittorrent (when people started choosing huge piece sizes), attackers would routinely do this, very cheaply-- just serve bogus random data. smaller chunks are very important here.

    the way i would approach what you describe (which is pretty cool) is that canonical should both:

    • provide the leaf hash with CIDv1. ipfs could address it but not easily transfer it (there may be exceptions here, where it could be a tunable parameter).
    • provide a new graph (chunked, dedup, etc), whose output matches the other leaf.
    • that way ipfs could pull the chunked one, and verify the whole one afterwards.

    you raise a good point though that we need to address how to look at objects and know whether to pull them when they may be too big.

  2. whyrusleeping commented on Aug 23, 2016

    @whyrusleeping
    Member

    Could this issue be moved to ipfs/notes please?

  3. robcat commented on Aug 31, 2016

    @robcat
    Author

    Just one tangential issue: when trying to add a big block, ipfs stops in the middle of the "add", like this:

    $ ipfs add -s size-1048577 /path/to/big/file
    779.62 KB / 2.42 MB [=========>-----------------] 31.49 % 0
    19:03:08.355 ERROR commands/h: object size limit exceeded client.go:247
    

    Maybe ipfs should refuse altogether to use a chunking size greater than BlockSizeLimit and/or give a better error message?

  4. changed the title [-]Clarify maximum block size[/-] [+]Give better errors around blocksize limitations[/+] on Sep 14, 2016
  5. changed the title [-]Give better errors around blocksize limitations[/-] [+]Clarify maximum block size[/+] on Sep 14, 2016
  6. ethernomad commented on May 30, 2017

    @ethernomad

    Is the blocksize limit enforced in the network layer, or just in the chunker?

  7. mateon1 commented on May 30, 2017

    @mateon1
    Contributor

    @ethernomad Both. As seen here, the chunker limits blocks to 1MB, but libp2p limits blocks to 2MB. The second limit would be hit for directories with large amounts of files prior to directory sharding being implemented.

  8. alphaCTzo7G commented on Aug 6, 2017

    @alphaCTzo7G

    I am guessing as network bandwidth, processor speeds, RAM sizes increases, the optimum block size should also scale as it becomes easier to do all the things that are mentioned in #3104 (comment). Further, the volume of data/size of individual files that people want to share with ipfs will also scale.

    Is there a way to increase/tune the blocksize for future? If not, wont the minimum block size become a bottleneck?

  9. Kubuxu commented on Aug 7, 2017

    @Kubuxu
    Member

    No, maximum blocksize won't become bottleneck as IPFS is not a PoW blockchain, there is no limitation on how many blocks can be sent per second.

    As @jbenet said: smaller blocksize allows for more deduplication and security but we are working on rising it either way to to characteristics of few version addressed ecosystems (git as one of the example) but it has to be done without any security trade off.

  10. Stebalien commented on Aug 11, 2017

    @Stebalien
    Member

    @alphaCTzo7G if you want to better understand this problem, take a look at this discussion on the IPFS discourse.

  11. whyrusleeping commented on Sep 3, 2017

    @whyrusleeping
    Member

    Closing this issue for now, please move further discussion to a new ipfs/notes issue, or discourse.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedSeeking public contribution on this issuekind/bugA bug in existing code (including security flaws)need/community-inputNeeds input from the wider communitystatus/readyReady to be worked

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions