-
@ mleku
2025-05-13 18:45:53
#realy #devstr #progressreport
i now have fully working full text indexers for notes and articles, and an index is now generated that makes searching for language tags on events fast
next task is to actually add the full text search NIP-50 functionality to the socket and http APIs, this will be a bit complicated, lots of set intersection operations
i was quite surprised how bleedin fast it was doing 117k events with the language tag, because it was decoding the binary events and then scanning the tags and it finished the whole lot in about 4 seconds... the binary codec on realy is really the fastest damn binary codec for nostr events evar... i mean, that's like decoding 27k events per second... the export has to then encode them into json and it takes about 20 seconds for the same lot... and marshaling json is basically just a string building operation, no complex logic to it.
if you are a #golang programmer and write code that does string processing, i can highly recommend the benefits of using bytes.Buffer and io.Reader/io.Writer model for screaming fast stream processing.