----------------------------------------------------------------------------

SnarfNews - USENET Transport Toolkit
by Alec Muffett

WWW: http://www.users.dircon.co.uk/~crypto/

E-mail: alecm@crypto.dircon.co.uk

----------------------------------------------------------------------------

Introduction

SnarfNews is a toolbox of utilities written mostly in Perl5 (with some shell
scripts), designed to help people manipulate USENET News traffic in useful
ways.

SnarfNews was initially written (back in 1990) in order to provide a
compressed, ASCII-fied USENET feed of a selection of newsgroups to a home
machine (an Amiga A500); the original project included a minimal NNTP server
and article-handling system, but that has since been dropped in favour of
INN on Linux, although it may be reborn one day if I have enough time.

Things you can do with SnarfNews, include:

   * Fetch news articles (by newsgroup) from multiple NNTP servers and
     inject them into a local news-spool via "rnews", "inews", POST or
     IHAVE.

   * Top-up your main newsfeed with articles from groups that you normally
     wouldn't get, by snarfing them from another server. This can be useful
     for receiving (eg:) local or national groups with poor propagation.

   * Put articles from a local NNTP server to a remote one, via NNTP's IHAVE
     or POST commands.

   * Gate e-mail lists into newsgroups. (preserves MIME attachments)

   * Gate newsgroups into e-mail lists. (preserves MIME attachments)

   * Automatically e-mail to yourself, articles that appear in specific
     newsgroups on an NNTP server.

   * Automatically save to disk any articles that are posted to a list of
     newsgroups on an NNTP server, for archiving or off-line news reading.

   * Transparently repost articles from one newsgroup on one NNTP server to
     another newsgroup on another NNTP server.

   * Collect articles from a NNTP server, batch and compress them, and
     e-mail them to a remote server for unbatching and injection into
     another newsfeed. (requires gzip)

   * Implement encrypted newsfeeds via batching/e-mail. (requires PGP 2.6.2
     or above)

   * Sanitise batched newsfeeds by removing excessively cross-posted
     articles, excessively large articles, "MAKE MONEY FAST", and other
     spams.

   * Inject traditional "rnews" batches into NNTP-only News servers, by
     translating them into individual IHAVE or POST commands.

   * Pull/push news through a SOCKS firewall. (requires "runsocks")

   * Implement "streaming" retreival of USENET articles, maximising
     utilisation of your network bandwidth and reducing connection time.

   * Enjoy automatic fall-back into RFC-977 compatability; ie: the software
     attempts to use common but non-standard NNTP etentions for speed, and
     falls back to stock NNTP commands if they fail.

----------------------------------------------------------------------------

Requirements

Things you will need to run SnarfNews include:

   * Access to a NNTP News server, preferably running INN.
   * perl5 (mandatory)
   * gzip (for compressed news batches)
   * pgp 2.6.2 or above (for encrypted news batches)
   * SOCKS software (for transport through SOCKS firewalls)

----------------------------------------------------------------------------

Overall design philosophy

Rather than create one great big program that could do everything, SnarfNews
is written as a toolkit; that is, it is made up of lots of smaller programs
which do particular tasks (some very similar), which get glued together in
order to provide whatever functionality is required.

Of course, this modularity means there is some loss in efficiency, but the
benefits for the user in being able to trivially extend the software, and in
the flexibility that this approach provides, far outweigh any overhead.

You will find that some functionality is duplicated, for instance much
functionality in "runfeeds" could be easily reimplemented using "ifbytes".
That's OK - as with most toolkits you'll soon find out that there is more
than one way to do things, to use the Perl motto.

SnarfNews deals in four basic types: the USENET article, the "rnews" batch,
the encoded "rnews" batch, and the encrypted "rnews" batch.

Hopefully you'll start to see a a common thread about this point.

SnarfNews' interpretation of an "rnews" batch is currently rather strict;
programs which expect to receive rnews batches cannot cope with being fed
bare articles, and vice versa. For those unfamiliar with rnews batches:
snarfnews slings groups of USENET articles around bundled together in a
format referred to as an "rnews batch".

The format is:

    #! rnews nnn
    [article headers]
    [article headers]
    [article headers]

    [article body]
    #! rnews nnn
    [headers of next article]
    ...
    ...

...where nnn is the size of the entire article (in bytes) measured from
where it starts on the next line. This is the standard format for grouping
articles together for transmission to other sites, and for a more detailed
reference the user is referred to the man page for "rnews".

Encoded (compressed and ASCII-fied) and encrypted batches are discussed
below.

----------------------------------------------------------------------------

Definition of the word "gronk"

Before we go on, we should define the word "gronk", which will turn up
rather frequently. Whilst coding snarfnews, the author began to get confused
by all the instances of the words "get", "put", "push", "pull", "merge",
etc, that were appearing in the code, and which meant different things in
different places.

To get around this, the author defined the word "gronk", thusly:

Gronk: to gather together a bunch of USENET articles collected from
disparate sources, and present them as a "rnews" batch.

Following from this, we have programs named "gronknntp", which gathers news
articles via NNTP and creates an "rnews" batch from then, and "gronkfiles",
which performs a similar task, reading its articles from files on disk.

----------------------------------------------------------------------------

Configuration

Well, since this is the first cut at producing a version of SnarfNews that
can be used by anyone (other than me) on any Unix workstation, it follows in
the great tradition of programmer's documentation writing that this should
be the skimpiest and least comprehensible part of the manual, so:

   * Unpack the software into a directory, eg: /usr/local/snarf

   * Hand edit the "snarf" script to reflect the directory name that the
     software is installed into, by setting the value of "SNARFHOME". Also
     take time to set the values of:

        o SNARFHOST
        o SNARFDOMAIN
        o SNARFADMIN
        o SNARFMAIL
        o SNARFXPOSTMAX
        o SOCKS_SERVER (if used)

     ...to reflect your local site requirements.

   * Run the "Configure" script; this will fix the paths to Perl5 in the
     appropriate scripts.

   * Run "make" to cleanup.

   * Finish reading the manual.

   * Edit the "conf/feeds" and "conf/filter/default" files appropriately.

I shall be working on a better way to do this, RSN.

When installing SnarfNews, check your permissions and consider who'll be
running it; if you intend to wire it into /etc/aliases for mail->news
gatewaying, the "$SNARFRUN" directory may have to be readable or writable by
"mail", "daemon" or similar.

Similarly, if you're heavily integrating it with "INN" and have not set up a
"news" group with group-execute bits, you may have to set permissions
likewise for the user that "innd" runs as.

To get around this in the first instance, I ship with the permissions 01777
on the "run" directory. If this is not suitable for you, change it, but be
aware of what may have to chenge to get it working in your environment.

----------------------------------------------------------------------------

Top Level Commands and Files

Makefile

     This is the makefile for "snarfnews" which does little other than help
     keep things tidy and set sensible file permissions, etc. Typing:

     % make clean

     or

     % make spotless

     - occasionally, is probably a good thing.

README.html

     ...is this file.

snarf program [arguments...]

     Snarf is the frontend for everything. Users are expected to invoke all
     the programs in the snarf suite as above, where the "snarf" script sets
     up essential environment variables, etc, which would otherwise clutter
     the environment unduly. Although this might seem awkward, it prevents
     "snarf" programs from leaking into the environment yet simplifies
     installation as there is only one script that needs any amount of
     serious work to configure.

     If a user-written script needs to use several tools from the snarf
     toolkit, it is probably simplest to write the script as you would
     normally, and then invoke:

     % snarf ~/bin/scriptname

     - or whatever, in the appropriate place.

----------------------------------------------------------------------------

Frontends

Users are encouraged to write their own frontends to the toolkit, but a few
pieces of standard functionality are supplied; where individual e-mail
messages or encoded/encrypted batches are being processed (the "mail2*" and
"cbatch2*"/"ebatch2*" commands), it is possible for them to be set up to
receive the traffic directly from "sendmail", for example by adding:

           snarf-inbound: "| /usr/local/snarf/snarf cbatch2rnews"

...to /etc/aliases, though care should be taken (especially by people
writing their own frontends) to ensure that there is no way that this
feature may be used by unauthorised people to meddle with your news traffic
or gain unauthorised entry to your system.

The supplied frontends are:

art2mail address [file]

     This command reads an article from "file" or from "stdin" and e-mails
     it to "address", eg:

     % snarf art2mail user@foo.co.uk < article

art2post servername [file]

     This command reads an article from "file" or from "stdin" and posts it
     in the NNTP server "servername" via NNTP's POST method.

batch2ihave servername [file]
batch2post server [file]

     These commands read an rnews-style batches of articles from "file" or
     from "stdin" and post them (individually) on the NNTP server
     "servername" via NNTP's IHAVE or POST methods, respectively.

batch2mail to-address [file]
batch2mailfrom to-address from-address [file]
batch2mailrepl to-address reply-address [file]

     These three commands read rnews-style batches of articles from "file"
     or from "stdin" and convert them to mail messages, which are then
     mailed to "to-address" by invoking "sendmail -t".

     The latter two commands permit the user to specify From: and Reply-To:
     addresses for the messages, respectively.

cbatch2mail to-address [file]
cbatch2rnews [file]

     These commands read a compressed and ASCII-encoded "rnews" batch from
     "file" or from "stdin" (see "batchencode", below) and either mail each
     article separately to "to-address", or pipe them into "rnews", which
     will load the articles into the local server.

     % cat batchfile | snarf cbatch2rnews

ebatch2mail keyfile to-address [file]
ebatch2rnews keyfile [file]

     These are analogues of "cbatch2mail" and "cbatch2rnews", but using
     PGP's IDEA encryption as implemented in "batchencrypt", below.

     The "keyfile argument is the name of a file to be found (only) in the
     "$SNARFSECRET" directory, containing a long text string which should be
     kept as secret as possible. The contents of this file will be used by
     PGP as a passphrase to conventionally encrypt the batch of articles
     before transmission to the remote site.

     The contents of the keyfile should be transferred securely to the other
     site (via PGP or floppy) and be installed there, where "ebatch2rnews"
     can be used to unpack it, and the result fed into "rnews" or
     "runbatch".

mail2post servername newsgroup [file]

     This command reads a e-mail message from "file" or "stdin", reformats
     it to be a USENET article, and posts it to "newsgroup" on "servername"
     using NNTP's POST method.

     One or more comma-separated newsgroups may be specified in the
     "newsgroup" argument, but there should be no spaces in the list.

----------------------------------------------------------------------------

Backends

b64decode [-debug] [file ...]

     This command reads base-64 encoded data from "stdin" or from filenames
     specified on the command line, and prints the decoded data on "stdout".
     Lines of input which do not look like plausible base-64 input are
     ignored, and can be viewed on "stderr" by specifying the "-debug"
     switch.

b64encode

     This command reads from "stdin" and reproduces its input on "stdout",
     using base-64 MIME encoding.

batchencode [file ...]
batchdecode [file ...]

     These commands read an "rnews" batch (or indeed any arbitrary data, but
     within "SnarfNews" they are used only for sending "rnews" batches) from
     "stdin" or filenames specified on the command line, and produce
     compressed, base-64 encoded output, bracketed by _begin and _end_
     statements.

     The batchencode process requires use of "gzip" specifically because of
     two important features of that program:

       1. Adaptive compression: "gzip" can cope sensibly with
          partially-compressed inputs, which is something that the standard
          "compress" program does not do well.

       2. Catenative input: "gzip" has a feature not present in standard
          "compress", that several files which have been independently
          compressed using "gzip" may be concatenated, and when gzcat'ed,
          the contents of all files will be correctly decompressed to
          "stdout".

batchencrypt keyfile [file]
batchdecrypt keyfile [file]

     These commands are analogues of "batchencode" and "batchdecode", above,
     but utilising PGP to compress, encrypt and ASCII-fy the input.

     The argument "keyfile" specifies the name of a file which will be
     searched for (only) in the "$SNARFSECRET" directory, and which contains
     a long line of arbirtary text which will be used as the passphrase for
     the encryption. (see "secret/example", below).

     Since PGP also uses adaptive compression and base-64 encoding, it is
     approximately equivalent to - sometimes slightly better than -
     "batchencode" in terms of the size of its output; however it has not
     been investigated whether PGP shares gzip's ability to cope with
     catenative input (see above), so for the moment it is recommended that
     the user does not build a system which would pipe more than one
     encrypted batch at-a-time, into a single instance of "batchdecrypt".

conva2a [-debug] [-skip-HEADER] [-die-HEADER] [-x-HEADER] ['Header: Value' ...]
conva2m [-debug] [-skip-HEADER] [-die-HEADER] ['Header: Value' ...]
convm2a [-debug] [-skip-HEADER] [-die-HEADER] ['Header: Value' ...]

     These programs provide essential filtering functionality for
     "SnarfNews" when gating USENET and E-mail together; the knowledgable
     user will already know that there are important differences between
     USENET postings and E-mail messages, specifically in what header fields
     are required for traffic to be propagated by either of those two means.

     These programs provide the ability to add, remove, and change arbitrary
     headers in a USENET article or E-mail message, and reformat the headers
     appropriately for re-transmission. Enough defaults are built into the
     filters to avoid most of the work in converting messages of one type to
     another, but certain data must be supplied on the command line. See (or
     use) the frontend scripts for examples.

        o "conva2m" expects to read a USENET article from "stdin" and
          reformats it as a E-mail message on "stdout". The user is expected
          to supply a recipient address on the command line, thusly:

          conva2m -skip-to "To: address"

        o "convm2a" expects a E-mail message and produces a USENET article.
          Notably, it checks the contents of the "From:" field; if the
          address in the "From:" field looks like a fully-qualified Internet
          mail address, it is left alone.

          If the address looks like:

          From: username@hostname

          ...it is modified to:

          From: username@hostname.$SNARFDOMAIN

          Otherwise, if it looks like a bare username, it is modified to:

          From: username@$SNARFHOST

          ...however, the user is free to override this by specifying:

          convm2a -no-from "From: user@host.domain.com"

          If these rules are not sufficent to provide the "From:" address
          rewriting functionality that you require at your site, users are
          recommended to write a wrapper to invoke "convm2a" with the
          "-skip-from" argument, and to insert their own "From:" header.
          Several pieces of USENET software (notably INN) will ot accept
          articles for posting which do not contain fully-qualified "From:"
          addresses.

        o "conva2a" expects a USENET article and modifies certain headers so
          that it can be re-posted on another NNTP server via NNTP's POST
          method; this should not be necessary if NNTP's IHAVE method is
          used instead.

     In all cases, a filter can be set to remove certain headers from the
     input via the "-skip-HEADER", eg: "-skip-subject" will remove the
     "Subject:" header. (header matching is case-insensitive)

     Similarly messages which contain "HEADER" will be silently discarded
     when "-die-HEADER" is specified; as a security mechanism this is
     permanently enabled for the header "X-Reflected-By:" to prevent
     messages from passing through more than one instance of convX2Y, which
     could otherwise lead to a disasterous USENET/E-mail loop.

     It is also for this reason that the "postart" and "ihaveart" scripts
     are designed to dilently ignore empty inputs, since they are likely to
     be the next in the pipeline after a convX2Y filter.

     "Conva2a" also has the ability to "comment" out any specified header
     using the "-x-HEADER" mechanism, which prepends "X-" to any given
     header; typically however, use of this will not be required, as USENET
     articles require very little modification for reposting beyond what
     "conva2a" provides by default.

     All filters can also add headers to the articles/messages that are
     produced, simply by specifying them on the command line, eg:

     conva2m 'X-My-Favourite-Cookie: Chocolate Chip'

     - will add the above header to the message generated by "convm2a".

     Important Note - users who are interested in making changes to E-Mail
     and USENET headers (other than those already available via the
     SnarfNews frontends) are strongly advised to read the reference
     material first (RFC822, RFC1036) to see what is/is not permitted.

     Never, under any circumstances, modify, add or remove a USENET
     article's "Message-Id:" header, if it already has one.

filterart [-debug] [configfile]
filterbatch [-debug] [configfile]

     These programs read a single article or an "rnews" batch from "stdin"
     (respectively) and only print articles to "stdout" (in their respective
     formats) if the article headers do not match any of the patters cited
     in the configuration file, the format of which is described below.

     If a configuration file is named on the command line (eg" "fred") it
     will be searched for in $SNARFCONF/filter ("$SNARFCONF/filter/fred")
     and then as an explicit path; if no such file is specified, the default
     ("$SNARFCONF/filter/default") is used.

     This permits the removal of MAKE MONEY FAST spams, excessively
     crossposted articles, traffic sourced from particular users/sites, and
     oversize articles before the traffic is forwarded to another process.

forprep

     This command copies from "stdin" to "stdout", indenting the output with
     a space character and a ">" sign.

gronkfiles [-debug]

     This program reads a list of filenames, one per line, from "stdin", and
     presents their contents as a "rnews" batch on stdout. Files specified
     as relative pathnames are taken from "$SNARFSPOOL".

gronknntp [-debug] [-save [-tree]] hostname[:port] [config]

     This program expects to find a list of newsgroups in the file "config"
     (default: $SNARFCONF/sites/hostname[:port]) and connects to the NNTP
     server on "hostname", where it tries to read new articles from each
     newsgroup, and present them as a rnews batch on "stdout".

     If the "-save" option is specified, articles are saved to individual
     files under the directory:

     $SNARFSAVE/the.group.name

     If "-tree" is also specified, articles are saved under:

     $SNARFSAVE/the/group/name

ifbytes command [arg ...]

     This command reads "stdin" up to an EOF and stores the input in a file;
     if the file's size is non-zero, then "command" is executed reading its
     "stdin" from the file, and the file is then removed.

     This command is chiefly used where "sendmail" (or similar) is at the
     end of a pipeline where it may (or may not) receive an empty input from
     "stdin"; when "ifbytes" is utilised in this case, "sendmail" will never
     actually be invoked, thereby preventing it from being confused.

ihaveart [-debug] [-genid] hostname[:port] [file]
postart [-debug] hostname[:port] [file]

     These command read a USENET article from "file" or "stdin" and post it
     on "hostname" using NNTP's IHAVE or POST methods, respectively.
     Alternative NNTP port numbers or service names can be specified by
     appending ":port" to the hostname.

     If "-genid" is specified with "ihaveart", and there is no "Message-Id:"
     header in the article, then a Message-Id: header will be created using
     the value of the "$SNARFHOST" variable as a hostname.

     Important: unless the "-debug" switch is specified, these two programs
     will accept and silently discard empty articles, so that no messages
     are generated on "stderr" when articles are dropped by "filterart" or
     the "convX2Y" scripts (below), thereby presenting an empty input to
     these programs.

innbatch [innfeedname ...]

     For interfacing with "INN"; assuming that the user has read/write
     permissions on the INN "out.going" directory and can invoke "ctlinnd",
     this script flushes the INN feed named "innfeedname", and invokes
     "gronkfiles" on the spooled articles to produce an rnews batch on
     "stdout".

     For reference, the INN "newsfeeds" entry for such a feed would look
     like:

     innfeedname[/exclude,exclude,...]::Tf:

     - see the INN "newsfeeds" man page for details.

lock [cookie]
unlock [cookie]

     These commands provide basic advisory locking functionality, if
     required. They are used by "runfeed" to prevent running of multiple
     instances of the same feed at the same time.


nntpmsgids [-debug] [-gronk] hostname[:port] [config]
gronkmsgids [-debug] hostname[:port] [pfcnt]

     These two commands are complementary and when combined by the user (or
     when one invokes the other - see below) they provide similar
     functionality to "gronknntp", although users may find additional uses
     for each, individually.

     The "nntpmsgids" is configured similarly to "gronknntp" (above) and
     expects to find a list of newsgroups in the file "config" (default
     filename as per "gronknntp") and connects to the NNTP server on
     "hostname", where it then produces a list of the Message-Ids of new
     articles in those groups, printing the list to "stdout".

     The same "SNARFXPOSTMAX" functionality of "gronknntp" is duplicated in
     this command.

     The "gronkmsgids" command reads a list of Message-Ids from "stdin",
     connects to the NNTP server on "hostname", retrieves the specified
     articles, and presents them on "stdout" as a "rnews" batch.

     The argument "pfcnt" (if specified) should be a small integer
     (typically between 1 and 3) which specifies a prefetch quantity, viz:
     the number of article-fetch requests which should queued so that
     article retrievals occur asynchronously, and the network connection is
     used to its maximal advantage.

     Combined with the asynchronous nature of the "nntpmsgids" command, this
     provides for exceptionally high throughput of article transferral, as
     there are few occasions where the link will ever be idle, eg: awaiting
     for the remote NNTP server to respond.

     Such a feed could be implemented via:

         stream \
             : nntpmsgids nntpserver | gronkmsgids nntpserver 2 | rnews

     ...in the "feeds" file, after setting up "$SNARFCONF/sites/nntpserver"
     - but it should be noted that this method of retrieval is of slightly
     higher risk than a standard "gronknntp" feed, since the article-count
     maintaining program ("nntpmsgids") cannot determine the successful
     connection of the article fetcher ("gronkmsgids").

     Moreover, if "gronkmsgids" fails to connect to your NNTP server for
     some reason and quits, you might lose articles which "nntpmsgids" will
     register as successfully retrieved (if it hasn't been killed by
     SIGPIPE).

     It is for this reason that "nntpmsgids" has an extra option, "-gronk".

     When invoked with the "-gronk" option, "nntpmsgids" will automatically
     launch "gronkmsgids" on the same host, with a "pfcnt" of two, and will
     pipe the list of message-ids directly to it, so that a feeds entry of:

         stream : nntpmsgids -gronk nntpserver | rnews

     - is functionally identical to the one previously cited, with the added
     advantage that "nntpmsgids" can detect a failure in "gronkmsgids",
     leading to better error recovery. In short: this is the preferred way
     to pull a feed using this method, though users may find creative uses
     for "nntpmsgids" and "gronkmsgids" separately.

     It should also be noted that "nntpmsgids" currently relies on the
     non-RFC977 "XHDR" command extension; if this is not available on your
     NNTP server, use "gronknntp" instead.

runbatch [-debug] "command"

     This program reads an rnews batch from "stdin", and for each article
     that it finds in the input, it invokes "command" (which can be a single
     command, or a shell pipeline), feeding the article into the command via
     "stdin".

     Obvious uses for this command include breaking open an rnews batch and
     reformatting and mailing the messages out to a maillist, via "conva2m"
     and "sendmail".

runfeed [-debug] [-nolock] [feedname ...]

     "Runfeed" is the driver script that can be used for many functions in
     SnarfNews. The "runfeed" configuration file is "$SNARFCONF/feeds", and
     contains lines of the format:

     feedname : puller [ : pusher ]

     Where:
        o "feedname" is a tag used to identify the command sequence
        o "puller" is a command or shell pipeline
        o "pusher" is another (optional) command or shell pipeline,
          separated from "puller" by a colon.

     Entries may be broken over several lines using backslash "\"
     characters.

     When a user invokes:

     % runfeed feedname

     ...the "runfeed" program searches its config file for an entry with a
     matching tag. If the matching entry contains no "pusher" command, then
     the "puller" command is executed and "runfeed" continues to search for
     other entries with the same tag.

     If a "pusher" command is defined, then the "puller" command is
     executed, with "stdout" redirected to a temporary file. When the
     "puller" finishes, if this file is NOT empty, then the "pusher" command
     is invoked, reading "stdin" from this file.

     Thus, the user can (for instance) use "gronknntp" to read articles from
     a NNTP server by specifying it as a "puller", and then mail the
     resulting rnews batch to another site by specifying a "pusher" to
     encode and mail the batch, the complete entry being:

     feedname \
             :gronknntp news.foo.co.uk \
             :batchencode | mail snarfnews@bar.co.uk

     Whereas, if you don't need to worry about sending out empty batches of
     articles (eg: you are fetching articles for your local feed) - you
     might prefer not to specify a "pusher" and just use a pipe:

     topup : gronknntp news.foo.co.uk | rnews -v

     The "-nolock" option disables locking of each feed in turn, which
     otherwise prevents multiple instances of the same feed being run
     simultaneously. Specifying no feedname on the command line will try to
     run all feeds listed in the "feeds" file.

timeout interval command args ...

     This program runs "command" for not longer than a specified "interval"
     of time (eg: "30s" = 30 seconds, "10m" = 10 minutes, "2h" = 2 hours) -
     after which time the process it killed using SIGTERM and then SIGKILL,
     if it has not already exited.

----------------------------------------------------------------------------

Miscellany

lib/snarf.pl

     Socket library functions for NNTP access.

conf/feeds

     Configuration file for "runfeed".

conf/filter/default

     Configuration file for "filterart". This file contains lines:

         # maximum number of groups an article can be crossposted to and accepted
         # (see also SNARFXPOSTMAX in "snarf" for gronknntp config of same)
         xpost 9

         # max article size specified as nnn (bytes) or nnnK (Kb) or nnnM (Mb)
         size 256K

         # filter out common spams; format:
         #     header [egrep pattern matching a spam]
         header ^subject:.*money.*money.*
         header ^subject:.*make.*money.*fast
         header ^subject:.*make.*cash.*fast
         header ^subject:.*\$\$\$

extras/inn-subscribe

     Weenie hack: reads a list of groupnames from "stdin" and invokes
     "ctlinnd" to create them. Saves typing it in from the command line
     every time.

secret/example

     Example "keyfile" for use in batch encryption. DO NOT USE THIS FILE!

     Instead, you should create your own "keyfile" by typing a long jumble
     of characters into a file, and transporting it via some secure means to
     the remote site.

     People running a recent version of Linux (kernel rev >2.0), or any
     other OS equipped with a good "/dev/random" random number generator,
     may like to try:

     % make asecret

     - which should produce a string suitable for redirecting into a
     "keyfile" in the "secret" directory; users on other operating systems
     will have to find a different way to create a high-entropy random text
     string.

----------------------------------------------------------------------------

FAQs

* Why doesn't "gronknntp" use NNTP's NEWNEWS, like "slurp" does?

The NEWNEWS command as described in RFC977 was ideal when NNTP was designed,
when USENET articles arrived in infrequent batches, and people using an NNTP
server were likely to be on the same local network. Clock granularity was
unlikely to be an issue, since on a LAN it is fairly easy to set all hosts
to be on the same time.

Nowadays articles arrive in their hundreds by the second, and people are
using long-distance comms to talk to NNTP servers; as such, there is no
guarantee that - especially if your NNTP server's clock is a few minutes
ahead of yours - that several dozen articles for your desired newsgroups
arrived in the interval when it was 12:00 noon on your workstation, and when
it was 12:00 noon on the NNTP server.

As such, since there is no easy way within NNTP to ask a server what time it
thinks it is, the simplest solution is instead to hack through the
newsgroups you desire, looking for new articles. This isn't so bad, really,
so long as your server supports the LISTGROUP command, as do all INN
servers.

* How do I get/put news through a SOCKS firewall?

After checking with your firewall administrator that it won't (a) get you
into trouble and (b) be blocked by access control lists, get a copy of
"runsocks" from the appropriate SOCKS software kit
(http://www.socks.nec.com/) and build it. Then set the value of SOCKS_SERVER
(in the "snarf" script) to point at your firewall SOCKS host, and modify
your "feeds" file to invoke "runsocks" before any puller or pusher that
needs to go through the firewall, eg:

    get-remote \
        : runsocks gronknntp news.remote.co.uk | rnews -v

    put-remote \
        : innbatch news.remote.co.uk \
        : runsocks batch2ihave news.remote.co.uk

...the SOCKS runtime library will be preloaded before execution of
"gronknntp", and so will transparently connect out through your SOCKS server
to the remote machine.

Users who are on machines which do not have a shared library capability (and
therefore cannot use "runsocks") will probably have to build their own
SOCKS-ified version of Perl5 and configure it into SnarfNews, in order to
achieve this functionality.

* How do I set up "SnarfNews" to gate between USENET and E-Mail?

The best way to describe this is probably by example; consider a mail-list
called "real-ale@foo.co.uk", which you want to gate to/from the newsgroup
"local.real-ale".

  1. create the newsgroup "local.real-ale" on your NNTP server.

  2. create an alias:

     real-ale-in: "|/usr/local/snarf/snarf mail2post servername local.real-ale

     ...and subscribe it to "real-ale@foo.co.uk".

     E-mail that arrives for "real-ale-in" will now be reformatted as USENET
     articles and posted to your local NNTP server running on "servername".

  3. Now do:

     % echo local.real-ale > $SNARFCONF/sites/real-ale

  4. ...and set up a feed entry in in $SNARFCONF/feeds:

     real-ale \
         : gronknntp localhost $SNARFCONF/sites/real-ale \
         : batch2mail real-ale@foo.co.uk

  6. Set up cron to run:

     /usr/local/snarf/snarf runfeed real-ale

     - every half hour. The cronjob will thus check for new articles posted
     to "local.real-ale", reformat them as mail messages, and post them to
     "real-ale@foo.co.uk".

Experimentalist sysadmins with control over their local INN daemon might
prefer to set up a "program" feed to support the "news to mail" half of the
functionality, above, along the lines of:

snarf!:!*,group.name:Tp:/usr/local/snarf/snarf art2mail address %s

IMPORTANT NOTE ON NETIQUETTE

Please exercise discretion if you choose to experiment with gating USENET
and E-mail back and forth.

Do remember that you are a small fish in a very big sea, and that however
"neat" you think doing this sort of thing may be, the readers of any
maillist are quite entitled to be thoroughly annoyed with you if you
propagate their posting onto USENET without their permission, or if (by
gating USENET postings back onto the maillist) you risk flood the maillist
with USENET spam and other junk.

Similarly, maillist moderators are quite entitled to be thoroughly annoyed
with you if random people start "posting" to the maillist through your
USENET gateway, when the whilst membership of the list is meant to be
"closed" or in some other way restricted.

In short: If you're in any doubt as to whenther you should do this sort of
thing, the answer should be don't do it.

----------------------------------------------------------------------------

Credits

SnarfNews is code bred from a variety of sources; "gronknntp" is a
second-generation descendent of the "pgnews" program by Jeffrey McGough.
Contributions to development have been made by Randal Schwartz, to whom
grateful thanks is directed for acting as a backstop for Perl queries.
Gracious thanks also to Larry Wall for Perl5.

The overall architecture of SnarfNews was developed whilst the author was in
a caffeine-induced alternative state of mind whilst working at the
University of Wales, Aberystwyth, as a systems programmer. Bear this in mind
if ever you visit Wales - it could happen to you too.

----------------------------------------------------------------------------

This file is part of SnarfNews Copyright (C) 1991,1992,1993,1994,1995,1996
Alec Muffett

This program is free software; you can redistribute it and/or modify it
under the terms of version 2 of the GNU General Public License as published
by the Free Software Foundation

This program is distributed in the hope that it will be useful, but WITHOUT
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
more details.

You should have received a copy of the GNU General Public License along with
this program; if not, write to the Free Software Foundation, Inc., 675 Mass
Ave, Cambridge, MA 02139, USA.

----------------------------------------------------------------------------
