16.02.25
Websites have no way to differentiate between bots or real human beings. The website has to load content to bot's and implement ever increasingly complex resource hungry or expensive defences that are ineffective. Bandwidth is being used up blocking legitimate clients and legitimate clients user experience is deteriorating.
I want to see the internet return to frictionless and brilliant tool for humanity to evolve. I am proposing a radical grassroots solution that does require changing the Internet's underlying infrastructure but it and makes the internet worse before it gets better. It requires collective community engagement and grassroots action which is a challenge to getting this idea off the ground. Before you reject this proposal out of hand give me a chance to demonstrate how this will work and show what problems it will solve.
I've always loved p2p systems and seeing the internet devolve has been painful to watch. In the process of coming up with this idea to stop bots I think I may have inadvertently fixed loads of other problems. I would like to make the disclaimer that this idea is not idiot proof and requires that each member of the community takes responsibility for their opsec (which may be this proposals downfall).
I've been down this rabbit hole many times before and I probably won't implement this idea in any meaningul way but I just wanted to show that it's probably possible to eliminate bots and scapers from a community of federated or disintermediated websites that work together. It is a closed system or "community overlay" but this will work. Getting people to join and adopt it is the bigger problem and it creates more splinters in the internet.
Other attempts at p2p communities do exactly the same thing and force users to create peers so it's no different in that respect. Having looked into this problem off and on and refined this idea I think that the current botnet style scrapers A.I companies are using are impossible to block without harming real people too.
I think that this idea might work better for a small community of early adopters initially rather than the entire internet so it's morphing into a safe community overlay of p2p/federated websites that work collectively to block DDOS, web scraping and other problematic web practices.
There are also interesting p2p projects like scuttlebutt that have overlapping ideas but I'm not sure if they suffer from bot's or how they solve that problem (if at all). If there was data within the platform that A.I companies wanted to scrape I imagine that the A.I company could just create a peer and scrape the entire system. Scuttlebutt is an offline first type replicated dataset so you download other peoples data and host it by design.
** Each person who is part of this new community creates a personal website
** They generate an private & public key-pair using asymmetric cryptography
** The personal website hosts the public key
** The public key is posted to a a federated reputation system along with metadata
::
Build a resilient federated network of reputation systems that receive public key pairs and URLs.
::
Website owners also generate a key pair and host a private/public key pair and send to the reputation services. The website owners implement a default block and rate limiting on all incoming requests.
::
When an actual human being wants to interact with a website they include their own public key. The request is signed using the public key and it includes an encrypted blob that is encrypted using the server's public key.
** The server does not implement IP blocking, captchas etc
** The server should be the only system that can decrypt the blob
** The server checks the client public key reputation
If the check passes the client can proceed to interact with the website free from captcha's, popups etc.
Let's say a DDOS attack is happening... each of the DDOS requests will be rejected causing bandwidth/load but hopefully manageable due to the rate limiting and the server returning a 4xx response and tiny payload (a message saying "bog off").
Legitimate clients should be let through. If there is a botnet machine that does not steal the private key of the victim it's possible to serve genuine requests to a hacked victim whilst simultaneously blocking the botnet DDOS attempt from the same IP.
To avoid peoples personal websites being DDOS'ed they also include this server side defence. They let through other valid users and the bot's used by the reputation systems to validate public key's. This bit gets a bit hairy because an attacker may try to game the reputation system by creating tonnes of website's and adding those websites to the reputation system.
If the reputation systems check DNS, time online, actual website domain names being used (blocking random IP's without certs) it will increase the cost to the hackers considerably. This will make it impossible for attackers to generate the huge number of fake clients and cause them considerable outlay and infrastructure. For each fake bot they will need a valid domain and reputation forcing them to fork out money for each bot client and risk that their bot's are discovered and blocked.
If attackers can steal the private key's of legitimate clients that can use those to continue their scanning. However if a client begins exhibiting crawling behaviour that a human being could never do. Say clicking links and traversing a website faster than a human being could do clicking or tripping up some other heuristic. Clicking on honeypot links or filling out honeypot forms their reputation could suffer and those hijacked clients could be rejected.
There are some challenges with the repudiation system and keeping it federated, avoiding centralised points of failure. However if we did adopt this system we would also have a federated network of websites that could also become a social media system that is actually p2p and federated. Using the underlying p2p aspects of the internet.
The personal website that hosts the public key can also host blog articles, posts and feeds. The repudiation system can include metadata allowing for other clients to discover content that is hosted in a truly federated & disintermediated way.
This will connect real people and give us a chance to be creative and solve real problems without sacrificing privacy. We can have a frictionless and nice internet experience again (at least within the community). We do not need to wait for some third party company or top down solution, we can trust and verify our own solution and build it all ourselves on top of existing infrastructure.
You force people to adopt and become part of the community before they can access the community. This is a real issue but I'm not sure if there is any other way around it. Instead of paywalls, popups and captchas you effectively now have a community wall. No matter how nice or good that community is it will still keep people out.
How do you keep bots and scrapers out ** and ** allow random people in at the same time... maybe you just can't unless the internet is fundamentally re-designed.
Hackers and the scraping botnet owners will still be looking for ways to get in, some actors and corporations will be looking for ways to "help" and then undermine the system. Especially if it makes search indexing or targeted ads more difficult.
Move over cookies, IP addresses and tracking pixels, we now provide cryptographic proof to advertisers that we are interested in something. - Mrkeen (hackernews comment)
Clients that interact with websites using their public key directly will leave a digital trail so let's say you add a chain of trust whereby you get a random public key as the server. The reputation system confirms it's linked to some real identity but doesn't tell the server which one. In the process of trying to solve many interconnected problems I'm probably generating a whole load of new problems and this is an astute point made by Mrkeen. I don't think it's fully addressed or solved ** yet ** and I appreciate the feedback!
This general requires consensus and people to work together harmoniously and solve real challenges in the face of real adversity. As I'm typing this sentence out I'm realising why this idea will never take off.
I believe that the internet is already federated and p2p in its original form. Ignoring the power that ISPs, domain registrars and website hosting providers have. Hosting your own personal website is federated. I know RSS was and is more federated than Bluesky or mastadon in my opinion. Bluesky requires centralised infrastructure and isn't disintermediated. I'm sure Bluesky has the power to mediate, moderate and intercept useful data.. in a truely federated or disintermediated system of nodes are on the same playing field. No one node knows about the entire system or has the ability to control or intercept data between other nodes.
Mastodon is a partially federated social media platform but it relies on servers that are managed and host peoples personal pages. The server owners can delete their server at anytime.
If we all have our own web servers that are secure and implement this kind of community protection overlay it eliminates the need for countless predatory industries, services and harm's that exist because the internet is now fundamentally broken and has been for a while now. Well the internet was always open for abuse relying on "good will".
This idea relies on the current internet infrastructure so it could be undermined by the current infrastructure.
If you need to interact with a reputation service of some kind for every new client connecting. That would require network calls and traffic correlating to the number of new inbound requests.
So you could DDOS a website by providing requests with junk public key's causing the server to consume resources checking whether those requests are genuine.
As a defense against scrapers and bots I think the idea has some merit. It may be replacing one form of DDOS for another if there is a targeted attack against the authentication bit.
A malicious reputation server could also potentially ok and allow bad requests if that aspect of the system isn't resilient. so reputation server's would also need a reputation system of some kind... and down the rabbit hole we go again.
It is unrealistic to think that the internet as a whole will adopt this overnight or at all but as an experiment to test it's I might create a prototype system to develop this idea further called jankynet or a different name.
I want to experiment with the idea of having each node acting as a reputation server to spread workload and refine the concept of a safe community overlay for the web that is better than mastadon, bluesky and other so called: "federated" social media platforms.
This is just the gist of the idea and you can probably think of improvements or idea's to enhance or laterally think of alternatives. I hope this is food for thought. having seen the long threads about bot defences, DDOS defences and the challenges website owners are up against providing decent user experience to human beings without being victimised by bot scrapers I think big problems require radical solutions.
We can break the internet even further for a bit as legitimate users are forced to create their own personal websites and cryptographic key pairs. But then the internet could potentially get better again. This system could also work over Tor for increased privacy and I feel that it's a relatively simple idea and concept.
Even though this idea is probably terrible the point remains... the solution we choose needs to be a grassroots effort and not something designed or proposed by the same actors that created these problems in the first place. It needs to be open source, free (as in freedom) and not beholden to outside influence. We probably can't solve the hard problems that face humanity until we fix the internet or create a new one as a pre-requisite. Unless your company thrives and lives off the complexity and innate problems the current internet has provided. Security professionals and vendors would be out of the job if we actually had a private and secure internet. Complexity serves's to make them "consultants" and "experts". I would argue being clever enough to solve pointlessly complex problems is not actually clever at all.
I think complex problems don't always require equally complex solutions.
You need to solve all the problems at the same time if the problems are all interconnected.