Repository navigation
Open
Labels
Description
For context, the long-term megalomaniac objectives (update Sep 2022):
- After completing the above, next item: Add tagging and update relevance ranking. Towards perfect metadata.
- De-duplication of search results.
- Also find non-matching info. Search for Linux, find items tagged Linux, Biggest Ubuntu swarm is shown first.
- Added to that is adversarial information retrieval for our Web3 search science. After above is deployed and tagging is added. Cryptographic protection of above info. Signed data needs to have overlap with your web-of-trust, unsolved hard problem.
- personalised search
- 3+ years ahead: row bundling
@arvidn indicated: tracking popularity is known to be a hard problem.
I spent some time on this (or a similar) problem at BitTorrent many years ago. We eventually gave
up once we realized how hard the problem was. (specifically, we tried to pass around, via gossip,
which swarms are the most popular. Since the full set of torrents is too large to pass around,
we ended up with feedback loops because the ones that were considered popular early on got
disproportional reach).
Anyway, one interesting aspect that we were aiming for was to create a "weighted" popularity,
based on what your peers in the swarms you participated in thought was popular. in a sense,
"what is popular in your cohort".
We deployed the first version into Tribler #3649 , after prior Master thesis research #2783. However, we lack documentation or specification of the deployed protocol.
Key research questions:
- What is the real deployed system behavior?
- What is the resource consumption?
- What is the accuracy and quality in general of the information?
- How can we attack or defend this IPv8 community?
Concrete graphs from a single crawl:
- messages and bandwidth in time
- hashes discovery and duplicates
- distribution of discovered popularity of swarms
- conduct swarm popularity check and compare results in real-time
- behavior of pub/sub mechanism for popularity feed
- dynamics of trust-based pub/sub auto-subscribe
Implementation of on_torrent_health_response(self, source_address, data)
ToDo @xoriole : document deployed algorithm in 20+ lines (swarm check algorithm, pub/sub, hash selection algorithm,handshakes, search integration, etc.).

Activity
xoriole commentedon Oct 4, 2018
Popularity Community

Introduction
Popularity community is a dedicated community to disseminate popular/live contents across the network. The content could be anything for eg. health of a torrent, a list of popular torrents or even search results. The way of dissemination of the content follows the publish-subscribe model. Each peer in the community is both a publisher and a subscriber. A peer subscribes to a set of neighboring peers to receive their content updates while it publishes its content updates to the peers subscribing it.
Every peer maintains a list of subscribing and publishing peers with whom it exchanges content. All contents from non-subscribed publishers are basically refused. Selection of peers to subscribe or to publish greatly influences the dissemination of the content both genuine and spam. Therefore, we try to select based on a simple trust score. Trust score indicates the number of times we have interacted with the node as indicated by the number of mutual Trustchain blocks. Higher the trust score better the chance of being selected (as publisher or subscriber).
Research questions
...
synctext commentedon Jul 2, 2019
ToDo:
describe the simplified top-N algorithm that is more light-weight (no pub/sub).
As-simple-as-possible gossip. Measure and plot 4 graphs listed above
61 remaining items
synctext commentedon Apr 6, 2023
New PR for torrent checking. After the release is done of Tribler 7.13 we can re-measure the popularity community. ToDo @xoriole. The new code deployment will hopefully fix issues and improve the general health and efficiency of popularity info for each swarm.
synctext commentedon Sep 4, 2023
Lets focus on the latest 7.13 release and re-measure. Can we find 1 million unique swarm in 50 days? How many long-tail swarms are we finding. Did we achieve the key goal of tracking 100k swarms for the {rough} popularity. If we succeeded we can move on and focus fully on bug fixing, tooling, metadata enrichment, tagging and semantic search.
@egbertbouman Measure for 24 hours the metadata at runtime we get from the network:
egbertbouman commentedon Sep 19, 2023
Just did a simple experiment with Tribler running idle for 24h in order to get some idea about which mechanisms in Tribler are primarily responsible for getting torrent information. It turns out that the vast majority of torrents are discovered through the lists of random torrents gossiped within the
PopularityCommunity.When zooming in on the first 10 minutes, we see that a lot of discovered torrents are discovered by the
GigaChannelCommunity(preview torrents). The effect of preview torrent quickly fades as preview torrents are only collected onup the discovery of new channels.While examining the database after the experiment had completed, it turned out that a little under 80% of all torrent were "free-for-all" torrents, meaning that they did not have a
public_key/signatureand were put on the network after a user manually requested the torrentinfo.ichorid commentedon Sep 19, 2023
Also, if you'd look at the top channels, they all were updated 2-3 years ago. That means their creators basically dumped a lot of stuff and then forgot about it, lose interest in it. But their creations continue to dominate the top, which is now a bunch of ๐ง ๐ง ๐ง
"Free-for-all" torrents gossip is the most basic form of collective authoring, and it trumped over the channels in real usage (e.g., search and top torrents) as @egbertbouman demonstrated above. This means that without a collective editing feature Channels are useless and misleading.
Overall, I'd say Channels must be removed (especially because of its clumsy "Channel Torrent" engine), and replaced with some more bottom-up system, e.g. collectively edited tags.
synctext commentedon Oct 3, 2023
๐ฒ ๐ฒ ๐ฒ
650 torrents are collected by Tribler in the first 20 seconds after startup?
Now we understand a big source of performance loss. Gigachannels is way too agressive in the first seconds and first minute of startup. No rate control or limit of IO or networking (32 torrents/second). Should be smoother and first rely on
RemoteSearchresults. Tribler will get unresponsive I believe on slow computers with this aggressive content discovery. Great work on the performance analysis! We should do that with all our features.No fix needed now! Only awareness that we need to monitor our code consistently for performance.
synctext commentedon Oct 10, 2023
Web3Discover: Trustworthy Content Discovery for Web3
@xoriole would be great to write a scientific arxiv paper on this, beyond traditional developer role. Also contributing to performance evaluation and scientific core. See IPFS example reporting, tooling, and general pubsub work plus "GossipSub: Attack-Resilient Message Propagation in the Filecoin and ETH2.0 Networks".
Problem description, design, Graphs, performance analysis, and vital role for search plus relevance ranking.
Timing: finish before Aug 2024 (end contract).
grimadas commentedon Aug 30, 2024
The problem remains relevant to this day. I see two main issues and possible directions for improvement:
Zero-trust architecture is expensive. If I only trust my own node, I need to check all torrents, which is a costly operation.
Solution: Enhance the torrent checker with reputation information. Include the source when you receive information about torrents checked by others. When verifying torrent health, cross-check with other reported sources. If the reported information is significantly inaccurate, downgrade the source's reputation. If the information seems "good enough," slightly increase the source's reputation.
This approach enables us to have torrent information enhanced with network health checks, not just checks from our own node.
Challenge: Torrent health is always dynamic data. We need a reliable estimation to determine what is significantly inaccurate and what is "good enough."
Biased health checks are not optimal. Currently, many peers perform redundant work by repeatedly checking the same popular torrents. Can we improve this process? Perhaps we could integrate information about who has already checked the torrents and how much we trust them.
Simplistic gossip mechanism: Randomly send information about 5 popular, checked torrents. While this could lead to repetition and over-redundancy, we need to demonstrate that this is indeed a problem (using a graph). With baseline metrics, we can identify opportunities for further improvement.
synctext commentedon Oct 30, 2026
Distributed Learning-to-Trust
Brainstorm: apply Learning-to-Trust in the popularity community to solve spam and attacks.
The popularity community remains highly scientific relevant. Trust is hard. Every time we make progress we discover that the problem is even harder then we envisioned (1999 information pools, Buddycast gossip 2005, BarterCast, MeritRank, unscented Kalman Filter 2024 etc). With LLM Voodoo techniques we can yet again make progress ๐ฅฒ
Key related work comes from Asia. Learning-to-trust: "Epistemic Context Learning: Building Trust the Right Way in LLM-Based Multi-Agent Systems".
All this work by the ML community is very naive, but making amazing progress. This theoretical model of attacks will fail when confronted with Internet reality. The PGP web of trust network designed around 1992 was attacked in 2019, see CVE-2019-13050. This was a simple poisoning attack (150000 spam signatures) . The entire public key pinning system was designed to be append-only, preventing unseen government tampering. Details: phd thesis 2002 and article Tolerating malicious gossip first published 2001. Attacker model is both very limited (corrupt relays), very extreme (e.g. anti-NSA overengineering), very naive attack-amplification-factor (
extremely large, extremely reliable, extremely censorship-resistant distributed filesystem which anyone can write to) (:Modern Keybase.io trusted crypto key management (went dead, central servers are never long-enduring). Historic note, once upon a time it was the prime directive of a 4 year post-doc (but was too early, too ambitious): MeritRank deployment for popularity community.

Learning-to-trust requirements: