This adds support for XOVER and some initial support for LIST OVERVIEW.FMT. The latter is specified by the RFC as "should implement when implementing XOVER".

This almostly makes nntpit work with the Pan newsreader--the message ids are still not to its liking (missing "@"). I've also got a patch for the latter already. Then it works just fine.

Sometimes while reading a post I’d like to view the “real thing” on Reddit. This commit adds an X-Reddit-URL header to all stories and comments that gives the content’s permalink.

When https://reddit.com/r/wallstreetbets is select as group, nntpit will report a lot of warnings, like:

[15] <- [GROUP wallstreetbets]

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json
...

but when you go to /r/programming, it will print a lot less warnings:

[15] <- [GROUP programming]

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json

** (process:2154): WARNING **: failed to parse subreddit json
[15] <- [HEAD 0]
[15] <- [HEAD 1]

I presume this has to do with number for posts in each thread? /r/wallstreetbets is overcrowded these days...

nntpit is started with: ./nntpit -D -t 3 -p 8119.

slrnpull [1] is a small tool that is used to pull news for offline read. Is there any way to get it working with nntpit?

Here is my slrnpull.conf sample:

emacs   100   0   0
linux   100   0   0

When I start nntpit with ./nntpit -D -t 3 -p 8119 and run slrnpull with slrnpull -h localhost:8119, I'm getting this in slrnpull output:

02/08/2021 19:59:05 slrnpull started.
02/08/2021 19:59:05 Reading /var/spool/slrnpull/slrnpull.conf
02/08/2021 19:59:05 Created dir /var/spool/slrnpull/news/linux.
02/08/2021 19:59:05 Connecting to host localhost ...
02/08/2021 19:59:05 Connected to host.  Posting ok.
02/08/2021 19:59:05 Fetching articles for emacs.
02/08/2021 19:59:06 ***listgroup emacs failed: Command not recognized
02/08/2021 19:59:06 Fetching articles for linux.
02/08/2021 19:59:22 ***listgroup linux failed: Command not recognized
02/08/2021 19:59:22 A total of 157 bytes received, 66 bytes sent in 17 seconds.

No articles were fetched.

[1] https://linux.die.net/man/1/slrnpull

The 'tin' newsreader requires the server to advertise READER mode. It also can't select groups it hasn't previously seen in the LIST ACTIVE response, even if you type in their names with the 'subscribe pattern' command. For this reason this commit adds a command line option to add new subreddits to the database. It only needs doing once.

LIST ACTIVE support also seems to make Thunderbird work.

Tested with tin 2.4.6 and Thunderbird 91.7.0

…eader.

The Pan newsreader has a sanity check that the message ids have chars out of <>@ in there.

nntpit didn't have @.

This patch adds @reddit to the message ids.

Thanks for making this, it works fine with Gnus too btw. Not sure how you feel about using Automake directly rather than a hand-written Makefile.in, but it avoids the awkward automake --add-missing # ignore errors step that's required at the moment to get install-sh.

A user on twitter reports that there is a compile error on Debian:

‘reddit_spool_store’: 
spool.c:70:9: error: void value not ignored as it ought to be
if (json_object_object_add(spool, id, object) == -1) {

I can't really figure out what it's warning about, and can't reproduce that problem. That value isn't void,

User says: gcc version 8.3.0 (Debian 8.3.0-6)

• 12: comma splice
• 16: oxford comma for clarity
• 24: bad subclause
• 30: comma splice
• 34: comma splice
• 46: removed oxford comma to join objects
• 48: add linking word to balance clauses
• 53: capitalize initialism
• 66: oxford comma to delineate
• 68: comma after introductory adverb - maybe ditch the intro
• 71: comma for flow
• 80: who's/whose
• 85: spelling
• 87: colon for sentence intro

Not sure what is a better place for such topic, so I guess I'll try here.

I did a similar project in the past, a reddit-to-nntp and lobsters-to-nntp gateway with posting support, and I've encountered two issues:

• Some NNTP software will actually slow down to a crawl when there is large number of messages returned by the NNTP server (I'm looking at Thunderbird, no such issues with slrn/gnus),
• There is an issue with read flag synchronization in case the user uses multiple machines. So, when I read a message, or catch up with a news group on one machine, I have to do the same on another machine.

I'm wondering if anyone has been considering using IMAP instead of NNTP as a gateway target? So reddit2nntp would become reddit2imap. That would definitely solve issue 2, but I'm wondering about 1. Some informative reddit threads are massive (e.g. r/OutOfTheLoop) and I wonder if e-mail clients would be prepared for such amounts of messages.

So, did anyone consider IMAP instead of NNTP on projects like these? Or have experience with IMAP with massive amount of e-mails? My IMAP inbox counts ~50.000 e-mails and it's fine, but 50.000 is peanuts when we're talking about mirroring interesting threads from reddit ;)

I'm a big fan of this project for the very reason described in the readme under the "Why?" section.

On a broader scope the Internet lost touch with itself on technical grounds now that the common population is thriving on using it.

I've been asking myself a question you may or may not care to think about but since you are prone to understanding my previous statement having made this tool I would like to throw it in the air around it; how do we bring back the good parts of the Internet that are becoming forgotten?

Let me know if you would be open to discuss this further, here or in private, my email is [email protected].

I sometimes see escaped html in comments, e.g. > it seems like we should be cleaning this up?

There are also markdown-style URLs, maybe we could be parsing that too.

Just making a note to investigate, it's mostly still readable.

This is neat; been meaning to write something like this for older systems. Curious how vintage NNTP clients handle it (since I'm curious about the retro applications), and so far, none of them like it:

• old Mozilla (but probably modern Thunderbird too) won't let you subscribe to groups that aren't listed already; if i use slrn to subscribe and add it to newsrc, it appears that it can see the metadata from GROUP, but can't request anything more; I don't see any further commands other than GROUP. No posts.
• Outlook Express suffers the same fate
• Xnews can see the group, but when it tries to get posts, it issues an XOVER command which nntpit rejects. It can at least add groups w/o them being listed.