Skip to content

Release v0.25.0 - #746

Merged
gammazero merged 42 commits into
releasefrom
release-v0.25.0
Dec 9, 2024
Merged

gammazero merged 42 commits into
releasefrom
release-v0.25.0

Conversation

@gammazero

Copy link
Copy Markdown
Contributor

Closes #745 (Release v0.25.0 issue)

lidel and others added 30 commits November 7, 2024 03:04
The dependency on goprocess is not needed by boxo, and removing it removes the need to support it in code dependent on boxo.

Closes #709
Remove dependency on goprocess
* feat: add instrumentation to delegated routing endpoint

* feat: allow passing in custom registry

* docs: update changelog

* chore: go mod tidy

* fix: create new registry unless set

* chore: adjust histogram buckets

* refactor: rename histogram

---------

Co-authored-by: Daniel N <2color@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat: add routing timeouts to delegated server

* chore: update changelog

* docs: CHANGELOG.md

---------

Co-authored-by: Daniel N <2color@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
#723)

pass context to server engine to register metrics
fix(routing/http/server): adjust bucket sizes for http metrics
#725)

fix(bitswap/client/providerquerymanager): don't end trace span until all providers are returned
…uting (#641)

Bitswap: refactor content provider elements

This PR performs a rather large and touchy refactor of things related to
Content providing and Content discovery previously embedded into Bitswap.

The motivations:

  * Make ProviderQueryManager options configurable

  * Align and separate coalesced layers: content routing must not be part of
  bitswap as in the future we will be using different exchanges (bitswap, http)
  for retrieval and content routing should be above exchange.

  * Align content routing interfaces with libp2p: to avoid crust, wrappers and
    user confusion, align Providers and Discovery types to
    libp2p.ContentRouting.

  * Reduce duplicated functionality: i.e. code that handles providing in
  multiple places and fails to take advantage of ProvideMany optimizations.

As a result:

  * ProviderQueryManager is now part of the routing module

  * A new providing.Exchange has been created

  * Bitswap initialization params have changed and Bitswap Network doesn't
    provide anymore (see changelog for more details)

Co-authored-by: Hector Sanjuan <code@hector.link>
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
* fix(bitswap/client/msgq): prevent duplicate requests

Previously, in-progress requests could be re-requested again during periodic rebroadcast.
The queue requests, and while awaiting response, the rebroadcast event happens.
Rebroadcast event changes previosly sent WANTs to pending and sends them again in a new message.

The solution here is to ensure WANT was in sent status for long enough, before bringing it back to pending.
This utilizes existing `sendAt` map which tracks when every CID was sent.

* Attempt rebroadcast more frequently than once per rebroadcast interval
---------

Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>
* Tests can signal immediate rebroadcast
  - No more need to update a synchronizd timer.
* Use zero interval for immediate rebroadcast
* simplify logic
Per ipfs/kubo#10593, if no one
is reading from the channel returned by RecursiveKeys() and the context is cancelled, streamIndex will hang indefinitely.

Proposed fix is to always select when attempting to write to the `out` channel. If the context is done and there is no one to read, we can abort.

Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
* Replace providerRequestBufferWorker with ChanQueue
* Set inProgressRequestStatuses map to nil when empty
* Remove setFindProviderTimeout and associated mutex. This can be set at creation time using WithMaxTimeout option.
* Replace forever-appended-to slice with circular buffer
* Increase default number of concurrent finds
* Optionally allow an unlimited number of concurrent find requests
- removes calls to prometheus.NewRegistry()
- replaces NewRegistry() call with global `prometheus.DefaultRegisterer`
  so by default boxo users who did not specify custom registry
  are not missing any metrics.
- ensures we don't panic if tests run in parallel and DefaultRegisterer is used
staticcheck fixes / remove ununsed variables
* Add debug logging for deduplicated queries
[skip changelog] bitswap/client: fix wiring when passing custom providerFinder
chore: fix 404 status URL

Signed-off-by: hishope <csqiye@126.com>
gammazero and others added 11 commits December 2, 2024 14:58
- Pass time.Duration instead of passing int that gets converted to time.Duration.
- Remove unneeded else clauses
- spelling
Set explicit options for bitswap default PQM, to make it more conservative and isolate it from a PQM without options
* feat(session): do not record erroneous session want sends

Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>
Recent improvements in bitswap should stabalize these tests.

Closes #327
* removed Startup function from ProviderQueryManager

Now `providerquerymanager.New` creates a `ProvicerQueryManager` that is already started.

There is no use case for starting PQM at a later time than it is created. Removing the need to call a `Statup` function separately from `New` is more convenient and reduces the opportunity for a problem if calling `Startup` is missed or if called multiple times.

* Remove flaky portion of test - requires synchronization to test sucessive timer delays
* no lifecycle context to shutdown ProviderQueryManager, use Close function instead.
* Use deque instead of slice for queues

Implementing a queue by appending to a slice will make more GC work as then end of slice's memory is reached. Instead use a deque that maintains a circular buffer that reuses memory from removed items.

* Reuse timer instead of creating one each loop iteration
@codecov

codecov Bot commented Dec 9, 2024 •

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 77.18041% with 191 lines in your changes missing coverage. Please review.

Project coverage is 60.43%. Comparing base (c71d535) to head (87c3914).
Report is 44 commits behind head on release.

Files with missing lines Patch % Lines
pinning/remote/client/client.go 0.00% 47 Missing ⚠️
...uting/providerquerymanager/providerquerymanager.go 85.76% 37 Missing and 7 partials ⚠️
bootstrap/bootstrap.go 39.62% 31 Missing and 1 partial ⚠️
bitswap/client/internal/session/cidqueue.go 14.28% 11 Missing and 1 partial ⚠️
routing/http/server/server.go 83.67% 8 Missing ⚠️
pinning/pinner/dspinner/pin.go 36.36% 6 Missing and 1 partial ⚠️
exchange/providing/providing.go 62.50% 4 Missing and 2 partials ⚠️
filestore/filereader.go 70.00% 4 Missing and 2 partials ⚠️
bitswap/client/client.go 93.44% 3 Missing and 1 partial ⚠️
...tswap/client/internal/messagequeue/messagequeue.go 91.30% 4 Missing ⚠️
... and 9 more

Impacted file tree graph

@@             Coverage Diff             @@
##           release     #746      +/-   ##
===========================================
+ Coverage    60.41%   60.43%   +0.02%     
===========================================
  Files          243      245       +2     
  Lines        31025    31063      +38     
===========================================
+ Hits         18743    18773      +30     
- Misses       10619    10622       +3     
- Partials      1663     1668       +5     
Files with missing lines Coverage Ξ”
bitswap/bitswap.go 69.13% <100.00%> (+1.69%) ⬆️
...client/internal/messagequeue/donthavetimeoutmgr.go 95.19% <100.00%> (ΓΈ)
bitswap/client/internal/peermanager/peermanager.go 91.85% <100.00%> (+0.12%) ⬆️
...tswap/client/internal/session/sessionwantsender.go 96.27% <100.00%> (+0.05%) ⬆️
bitswap/client/wantlist/wantlist.go 90.90% <ΓΈ> (-0.88%) ⬇️
bitswap/network/connecteventmanager.go 88.54% <100.00%> (+2.08%) ⬆️
bitswap/server/internal/decision/engine.go 92.15% <100.00%> (+0.71%) ⬆️
bitswap/testinstance/testinstance.go 86.44% <100.00%> (+0.23%) ⬆️
bitswap/testnet/peernet.go 38.46% <100.00%> (-4.40%) ⬇️
blockservice/test/mock.go 100.00% <100.00%> (ΓΈ)
... and 32 more

... and 5 files with indirect coverage changes

@gammazero gammazero added the release Merging this PR will create a tagged release label Dec 9, 2024
@github-actions

github-actions Bot commented Dec 9, 2024

Copy link
Copy Markdown

Suggested version: 0.25.0

Comparing to: v0.24.3 (diff)

Changes in configuration file(s):

diff --git a/go.mod b/go.mod
index 54cb060e..3cb9d498 100644
--- a/go.mod
+++ b/go.mod
@@ -12,6 +12,8 @@ require (
 	github.com/cskr/pubsub v1.0.2
 	github.com/dustin/go-humanize v1.0.1
 	github.com/gabriel-vasile/mimetype v1.4.6
+	github.com/gammazero/chanqueue v1.0.0
+	github.com/gammazero/deque v1.0.0
 	github.com/gogo/protobuf v1.3.2
 	github.com/google/uuid v1.6.0
 	github.com/gorilla/mux v1.8.1
@@ -37,10 +39,9 @@ require (
 	github.com/ipld/go-car/v2 v2.14.2
 	github.com/ipld/go-codec-dagpb v1.6.0
 	github.com/ipld/go-ipld-prime v0.21.0
-	github.com/jbenet/goprocess v0.1.4
 	github.com/libp2p/go-buffer-pool v0.1.0
 	github.com/libp2p/go-doh-resolver v0.4.0
-	github.com/libp2p/go-libp2p v0.37.0
+	github.com/libp2p/go-libp2p v0.37.2
 	github.com/libp2p/go-libp2p-kad-dht v0.27.0
 	github.com/libp2p/go-libp2p-record v0.2.0
 	github.com/libp2p/go-libp2p-routing-helpers v0.7.4
@@ -50,14 +51,15 @@ require (
 	github.com/mr-tron/base58 v1.2.0
 	github.com/multiformats/go-base32 v0.1.0
 	github.com/multiformats/go-multiaddr v0.13.0
-	github.com/multiformats/go-multiaddr-dns v0.4.0
+	github.com/multiformats/go-multiaddr-dns v0.4.1
 	github.com/multiformats/go-multibase v0.2.0
 	github.com/multiformats/go-multicodec v0.9.0
 	github.com/multiformats/go-multihash v0.2.3
-	github.com/multiformats/go-multistream v0.5.0
+	github.com/multiformats/go-multistream v0.6.0
 	github.com/polydawn/refmt v0.89.0
 	github.com/prometheus/client_golang v1.20.5
 	github.com/samber/lo v1.47.0
+	github.com/slok/go-http-metrics v0.12.0
 	github.com/spaolacci/murmur3 v1.1.0
 	github.com/stretchr/testify v1.9.0
 	github.com/whyrusleeping/base32 v0.0.0-20170828182744-c30ac30633cc
@@ -73,6 +75,7 @@ require (
 	go.opentelemetry.io/otel/trace v1.31.0
 	go.uber.org/multierr v1.11.0
 	go.uber.org/zap v1.27.0
+	golang.org/x/exp v0.0.0-20241009180824-f66d83c29e7c
 	golang.org/x/oauth2 v0.23.0
 	golang.org/x/sync v0.8.0
 	golang.org/x/sys v0.26.0
@@ -97,7 +100,6 @@ require (
 	github.com/go-logr/stdr v1.2.2 // indirect
 	github.com/go-task/slim-sprig/v3 v3.0.0 // indirect
 	github.com/godbus/dbus/v5 v5.1.0 // indirect
-	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
 	github.com/google/gopacket v1.1.19 // indirect
 	github.com/google/pprof v0.0.0-20241017200806-017d972448fc // indirect
 	github.com/gopherjs/gopherjs v0.0.0-20190430165422-3e4dfb77656c // indirect
@@ -118,6 +120,7 @@ require (
 	github.com/ipfs/go-verifcid v0.0.3 // indirect
 	github.com/jackpal/go-nat-pmp v1.0.2 // indirect
 	github.com/jbenet/go-temp-err-catcher v0.1.0 // indirect
+	github.com/jbenet/goprocess v0.1.4 // indirect
 	github.com/klauspost/compress v1.17.11 // indirect
 	github.com/klauspost/cpuid/v2 v2.2.8 // indirect
 	github.com/koron/go-ssdp v0.0.4 // indirect
@@ -166,7 +169,7 @@ require (
 	github.com/prometheus/common v0.60.0 // indirect
 	github.com/prometheus/procfs v0.15.1 // indirect
 	github.com/quic-go/qpack v0.5.1 // indirect
-	github.com/quic-go/quic-go v0.48.1 // indirect
+	github.com/quic-go/quic-go v0.48.2 // indirect
 	github.com/quic-go/webtransport-go v0.8.1-0.20241018022711-4ac2c9250e66 // indirect
 	github.com/raulk/go-watchdog v1.3.0 // indirect
 	github.com/stretchr/objx v0.5.2 // indirect
@@ -183,7 +186,6 @@ require (
 	go.uber.org/fx v1.23.0 // indirect
 	go.uber.org/mock v0.5.0 // indirect
 	golang.org/x/crypto v0.28.0 // indirect
-	golang.org/x/exp v0.0.0-20241009180824-f66d83c29e7c // indirect
 	golang.org/x/mod v0.21.0 // indirect
 	golang.org/x/net v0.30.0 // indirect
 	golang.org/x/text v0.19.0 // indirectdiff --git a/examples/go.mod b/examples/go.mod
index ac7a5343..b5856dee 100644
--- a/examples/go.mod
+++ b/examples/go.mod
@@ -11,8 +11,7 @@ require (
 	github.com/ipfs/go-datastore v0.6.0
 	github.com/ipld/go-car/v2 v2.14.2
 	github.com/ipld/go-ipld-prime v0.21.0
-	github.com/libp2p/go-libp2p v0.37.0
-	github.com/libp2p/go-libp2p-routing-helpers v0.7.4
+	github.com/libp2p/go-libp2p v0.37.2
 	github.com/multiformats/go-multiaddr v0.13.0
 	github.com/multiformats/go-multicodec v0.9.0
 	github.com/prometheus/client_golang v1.20.5
@@ -44,6 +43,8 @@ require (
 	github.com/flynn/noise v1.1.0 // indirect
 	github.com/francoispqt/gojay v1.2.13 // indirect
 	github.com/gabriel-vasile/mimetype v1.4.6 // indirect
+	github.com/gammazero/chanqueue v1.0.0 // indirect
+	github.com/gammazero/deque v1.0.0 // indirect
 	github.com/go-logr/logr v1.4.2 // indirect
 	github.com/go-logr/stdr v1.2.2 // indirect
 	github.com/go-task/slim-sprig/v3 v3.0.0 // indirect
@@ -95,6 +96,7 @@ require (
 	github.com/libp2p/go-libp2p-kad-dht v0.27.0 // indirect
 	github.com/libp2p/go-libp2p-kbucket v0.6.4 // indirect
 	github.com/libp2p/go-libp2p-record v0.2.0 // indirect
+	github.com/libp2p/go-libp2p-routing-helpers v0.7.4 // indirect
 	github.com/libp2p/go-msgio v0.3.0 // indirect
 	github.com/libp2p/go-nat v0.2.0 // indirect
 	github.com/libp2p/go-netroute v0.2.1 // indirect
@@ -109,11 +111,11 @@ require (
 	github.com/mr-tron/base58 v1.2.0 // indirect
 	github.com/multiformats/go-base32 v0.1.0 // indirect
 	github.com/multiformats/go-base36 v0.2.0 // indirect
-	github.com/multiformats/go-multiaddr-dns v0.4.0 // indirect
+	github.com/multiformats/go-multiaddr-dns v0.4.1 // indirect
 	github.com/multiformats/go-multiaddr-fmt v0.1.0 // indirect
 	github.com/multiformats/go-multibase v0.2.0 // indirect
 	github.com/multiformats/go-multihash v0.2.3 // indirect
-	github.com/multiformats/go-multistream v0.5.0 // indirect
+	github.com/multiformats/go-multistream v0.6.0 // indirect
 	github.com/multiformats/go-varint v0.0.7 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
 	github.com/onsi/ginkgo/v2 v2.20.2 // indirect
@@ -145,7 +147,7 @@ require (
 	github.com/prometheus/common v0.60.0 // indirect
 	github.com/prometheus/procfs v0.15.1 // indirect
 	github.com/quic-go/qpack v0.5.1 // indirect
-	github.com/quic-go/quic-go v0.48.1 // indirect
+	github.com/quic-go/quic-go v0.48.2 // indirect
 	github.com/quic-go/webtransport-go v0.8.1-0.20241018022711-4ac2c9250e66 // indirect
 	github.com/raulk/go-watchdog v1.3.0 // indirect
 	github.com/samber/lo v1.47.0 // indirect

gorelease says:

gorelease: preparing to load packages for github.com/ipfs/boxo: looking for missing dependencies: go: downloading github.com/ipfs/boxo v0.24.0
go: downloading github.com/ipfs/boxo v0.24.3
go: gorelease-load-module imports
	github.com/ipfs/boxo/exchange/providing: cannot find module providing package github.com/ipfs/boxo/exchange/providing
go: gorelease-load-module imports
	github.com/ipfs/boxo/routing/providerquerymanager: cannot find module providing package github.com/ipfs/boxo/routing/providerquerymanager

gocompat says:

HEAD is now at 980b6696 Merge pull request #713 from ipfs/release-v0.24.3
Previous HEAD position was 980b6696 Merge pull request #713 from ipfs/release-v0.24.3
Switched to branch 'main'
Your branch is up to date with 'origin/main'.

Automatically created GitHub Release

A draft GitHub Release has been created.
It is going to be published when this PR is merged.
You can modify its' body to include any release notes you wish to include with the release.

@gammazero
gammazero marked this pull request as ready for review December 9, 2024 19:56
@gammazero
gammazero requested a review from a team as a code owner December 9, 2024 19:56

@lidel lidel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gammazero
gammazero changed the base branch from main to release December 9, 2024 21:27
@gammazero
gammazero merged commit 03edbcd into release Dec 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release Merging this PR will create a tagged release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Release 0.25.0

9 participants