the scrape

im personally responsible for the original scrape of the private ebooks tracker bibliotik that eventually was used to produce books3, one of the most widely used and controversial llm training datasets

@frozenaesthetic

Share a piece of lore about yourself pic.twitter.com/NX4QIsJjKD

for those who don't know bibliotik was a private community that had hundreds of thousands of retail quality ebooks totally unavailable anywhere else, mostly downloaded from libraries and then with stripped DRM

they had a week long neutral-leech event. it was during covid and i was barely doing jack shit at work. i knew what i had to do.

grinded for basically the whole week straight, took like 50 transmission dockers and a fuckload of custom scraping software and my nonsensically overpowered NAS, but made it through with like 24 hours to spare

then i passed it off to the archivist for publication, wanted to make sure it was made widely available without catching flack

i got banned anyway of course, private trackers don't take kindly to public redistribution of their content

and the rest is history. just always funny when i see it brought up in news articles and im like damn that was literally all downstream of me taking too much adderall that one week

to everyone suggesting i should be more careful posting about a potential felony, this was all in early 2020 so we finally past the statute of limitations 👊