/admin/spam system

part of the ArsDigita Community System by Philip Greenspun

User directory: none
Admin directory: /admin/spam/
Data model: subsection within /doc/sql/spam.sql
Procedures: within /tcl/spam-daemon
Related modules: automatic email processing, bulkmail and qmail configuration

The Big Picture

This is a module to be used only by the publisher. Moreover, it has very little user interface of its own: URLs underneath /admin/spam are typically only invoked from within /admin/users. The data model is in /doc/sql/spam.sql. However, there is a facility for posting scheduled, periodic alerts or mailings, which has an interface under /admin/spam.

Under the Hood

we keep a history of all the spam we've sent to users in the spam_history table
forms that allow a publisher to spam (e.g., /admin/users/action-choose) generate a new spam_id for the blank form; this way a double click does not result in a spam being sent twice (spam is bad enough!)
The spam sending system tries to send email using each users email_type preference (from the users_preferences table). You can enter copies of the spam mail message formatted as plain-text, HTML, and AOL-HTML (some HTML subset). The spam sending routine will attempt to send the proper format to each user based on their preference. The default is plain text.

Sending spam from the browser

The /admin/spam page has a link to send a plain spam message to a class of users, or to send a combined plain and HTML message. In both cases you will be given a form which asks for a subject line, a target class of users, and a date to send the message. The from-address will probably be overridden with a machine-generated email address in order to make the automated bounce handling scripts work.

General Tcl Substitution in Message Body and Subject Line

The spam entry forms all have a checkbox labeled Template?.

If checked, then Tcl evaluator will be run at the time the message is actually sent on message subject and body, substituting variables or procedure calls wherever \$ and \[\] chars are found. This is specially useful for creating automatically generated templates, such as a daily report which runs a tcl procedure to query the database or create some other summary message.

Note: if you have the Template? option selected, make sure you use backslash to quote any $ or '[]' characters in your message, if you do not want them to undergo evaluation by the Tcl parser.

The following variables are guaranteed to be set in the environment, for use by your template.

db2 a database handle for any subqueries you want to make
user_id the user_id of the current target user
email the email address of the current target user
first_names
last_name

Guessing user's email preferences

In the site .ini file, you can set up a list of patterns to be used to guess the email type preference of new users, based on their email address.

The param is an association-list, with SQL patterns on the left, and pseudo-mime types on the right. Supported types right now are text/html, text/html, and text/aol-html.

EmailTypes={%@hotmail.com text/html} {%@aol.com text/aol-html}

Manually Sending Spam From a File on the Server

You can schedule a spam which gets its content from files in the file system, using the "Post spam from file" link on the admin spam page. It will look n the "Drop Zone" directory (described below) for files with the specified names, and send the spam using them as the content to the target user class of your choice.

Spam Which Is Sent Automatically: Newsletter Groups

The spam system will periodically check a "drop zone" directory for uploaded files. You can associate a filename prefix with a scheduled spam message, so that every time a new file is uploaded, it is scheduled to be sent out to a particular target class of users. The system can check either for a file containing static content, or for a template file which contains Tcl code that is executed to generate the message body. The template option is useful for generating periodic status reports or alerts.

Below is an example of the periodic spam admin form, showing the options for configuring periodic messages which get their content from named files the drop-zone directory:

Daily Spam File Locations
Your Workspace : Admin Home : Spam : List Daily Spam Files

Spam files to look for in drop-zone directory "/web/arsdigita/spam/".
To delete an entry, just enter an empty string for the filename and subject, and press the Modify button.
'From address' is optional; if left blank, the default spam system from-address will be used.

Periodic Email (Spam) Entries

User Class Subject Filename

User Class
Subject

Filename

From Address

Period Day of week Day of month
Template?

User Class
Subject

Filename

From Address

Period Day of week Day of month
Template?

User Class
Subject

Filename

From Address

Period Day of week Day of month
Template?

User Class
Subject

Filename

From Address

Period Day of week Day of month
Template?

User Class
Subject

Filename

From Address

Period Day of week Day of month
Template?

User Class
Subject

Filename

From Address

Period Day of week Day of month
Template?

Modify Spam Entries submit button goes here

Add New Periodic Email Entry

User Class

Subject

Filename

From Address

Period Day of week Day of month

Template?

Add New Entry button goes here

You can enter the following information for an automatic spam daily message:

User Class: (pulldown menu)
Subject: Note that you can include the current date in the subject line of the spam, by including the string "%%DATE%%" in the subject.
File Prefix: The filename prefix where you will deposit the new content for periodic mailings.
Period: How often the message should be sent. This only makes sense for template files, i.e., where the message content is generated automatically on demand. If the template? option is selected, then the named file will be used as a Tcl template.
If you are supplying "static" content by uploading new files to the dropzone with the date conventions described in the next section, then leave the default setting of Daily for the period.

Drop Zone Directory

The files should be placed in the "Drop Zone" directory specified by the .ini parameter DailySpamDirectory:

Example:

DailySpamDirectory=/web/eplay/spam

For each spam defined in the Daily Spam list, the system will look for the following files:

file_prefix-MM-DD-YYYY
file_prefix

Note: Be sure to always use two-digits for Month and Day fields, i.e., 03-06-1999. Don't forget the leading zeros.

If a file whose name matches with the specified prefix and the current day's date is found, the contents are queued up to be sent as spam to the designated target user class.

The spam system will only send a dated spam once. It keeps a history of all spams sent, and will be robust across server restarts. If the server is restarted in the middle of sending a spam, the spam daemon will resume sending where it left off in the list of users.

Be very careful with filenames that have no date suffix!

If you use a filename with no date suffix, the spam will be sent once a day from the file. This behavior is designed to support a special case spam for new users, where the user class contains a magic query like

select user_id from users where trunc(registration_date)=trunc(sysdate-1)

which is carefully constructed to always select a mutually exclusive set of users each day, and never repeat the same user twice.

HTML and AOL content types

Some users will have designated preferred MIME types for email via the users_preferences table. Currently we support HTML and AOL types in addition to plain text. If you create auxiliary files with the names

file_prefix-html-MM-DD-YYYY
file_prefix-aol-MM-DD-YYYY

Then content from these files will be sent preferentially to users who have their email types preferences set in the database.

Internal API for other modules to post scheduled spam messages to a user or group of users

If another module in the ACS wishes to post a spam message, they should use the following interface function:

# insert a new spam message into the spam queue, returns the spam_id of the new message
spam_post_new_spam_message {{
          -db 0 
          -template_p "f"
          -from_address ""
          -title ""
          -body_plain ""
          -body_html ""
          -body_aol ""
          -target_users_description ""
          -target_users_query ""
          -send_date "sysdate"
          -creation_user ""
          }}

The target_users_description should be an English description of who the designated target users are, for administrative documentation purposes.

The target_users_query should be a SQL query which returns the user_id for each user you want to send email to. Example

select user_id from users where last_name = 'Kennedy'

Example SQL Queries for spam messages

You supply a SQL query which will be used to generate the target list of recipients for your spam message. Your query should return a single column, user_id. The spam system will automatically join this with the users_spammable view, which is a view that filters out users who have elected not to receive email from the system. So don't worry about pre-filtering your query via the users_spammable view. Just supply SQL which coughs up a list of user_ids.

A simple plain text spam to members of a specific user_group:
select u.user_id
from users u, user_group_map ugm 
where ugm.group_id = 12345
and ugm.user_id = u.user_id
A query which gets all users with an email address at aol.com.
select user_id
from users  
where email  like '%@aol.com'

Setting the .ini Parameters

Here is a summary of the .ini params you will need

 
[ns/server/yourserver/acs/spam]

; Pairs of {email_addr_pattern pseudo-mime-type}
EmailTypes={%@hotmail.com text/html} {%@aol.com text/aol-html}
DailySpamDirectory=/web/yourserver/spam
SpamRobotFromAddress=email-robot@yourdomain.com

BulkMail

By default the spam system uses the built-in AOLserver ns_sendmail routine. This is adequate for low volume mailings, but if you need to send mail to more than about 500 users, then it has serious limitations. A high-performance module called bulkmail which has a multi-threaded mail client which can talk to multiple mail servers concurrently, and can generate the QMAIL style VERP return addresseses to make automatic bounce handling possible. This will soon be available as part of the core ACS distribution, and the spam module will have an option to use this as the mail transport rather than sendmail.

For more info on the configuration of bulkmail and qmail, see bulkmail and qmail configuration

Utilities

There are some useful utilities defined by the spam and email handling modules:

send_email_attachment_from_file allows you to send a file via ns_sendmail as a MIME encoded email attachment. The file can have binary content such as a JPEG image or PDF document.

philg@mit.edu

hqm@arsdigita.com