Reports Server

1. Objectives
   
   1. Stable location for reports that survives the wiping of /home/nsadmin or /web/client-content directories.
   2. Ability to configure reports for a new client in under 10 minutes.
   3. Ability to store and retrieve easily all reports generated for the client in its history.
   4. Sharing of the reverse DNS cache over all clients.
   5. Ability to separate the Reports Server into a separate box if necessary.
   
   
2. Set-up (for AOLserver):
   
   1. compile analog for your machine, making necessary changes to analhead.h and analog.cfg to reflect your directory structure.
   2. change configure-new-service-2.tcl to set cfg_group to the system group which should be able to edit the new config files. Make sure that the appropriate chmod is available to nsadmin.
   3. change generic.cfg to reflect your directory structure and desires.
   4. Make sure your AOLserver's .ini file has (in [ns/server/servername/module/nslog]):

      RollFmt=%Y-%m-%d-%H:%M
      RollHour=0

      If you want another format, you'll have to edit daily-report-procs.tcl and alter the string name "archive_log_name".

   5. Edit defs.tcl to match your maintainer email and directory structure.
   
   
3. File Structure:
   A list of services resides in /web/reports/servers.txt. This is edited by the automated configuration form. This file is reference whenever the scheduled reports are run, and it contains the server name, a space, and a contact email followed by a newline.

   Each service has a separate directory under the Reports Server Root, e.g., /web/reports/arsdigita for the arsdigita.com server and /web/reports/edf for the Scorecard server. The name of this directory should be the same as that pf the server. Each service directory contains the following:

   • service-name.cfg file (e.g., "khaki.cfg") for analog, modified from /web/reports/generic.cfg.
   • Optionally, you may have any of the following: a service-name.weekly.cfg, a service-name.monthly.cfg, and a service-name.yearly.cfg. If you have any of these, Reporte will use these files to determine report structure for those reports, respectively.
   • directories named with four-digit years, e.g., "1998"; these contain analog reports for each day of the year, named "YYMMDD.html" (e.g., "980928.html" for September 28, 1998). These also contain weekly reports named for the date of the week beginning (always on Mondays), e.g. "981116.week.html". Monthly reports are named "YYMM.month.html". These directories are created dynamically as needed.
   • cache, a directory for storing information that analog needs to build cumulative reports
   • index.tcl, which is the user page. Offers links to the past seven days' daily reports, the last four weeks' weekly reports, and the last month's report. It also contains a tail of today's raw log file, and the directories of years. The year directories won't have index.html files; we'll just let AOLserver deliver them (they'll come out in chronological order anyway thanks to the choice of file names above).
   The DNS cache resides at /web/reports/analod/dnsfile.txt and is used by all the servers. It trusts lookups for 4 weeks.

   All files must be writable by nsadmin.

   
   

4. Cron Structure:
   Every night at 2 a.m., an AOLserver ns_schedule_daily procedure will start processing logs from /home/nsadmin/log/. Analog will be exec'd by Tcl and fed the appropriate file names. Following processing, the same script will gzip the log in the /home/nsadmin/log/ directory should the log have been rolled. If any errors occur with the scheduled generation, the contact email is notified. Note that rolled logs must have the format "server-name.log.YYYY-MM-DD-00:00".

   The procedure will also run at 10am and 6pm to verify that the reports have been generated, and if they have not, to generate them.

   
   

5. Running Manually:
   If necessary, you can force the reports server to generate it's reports by invoking the following procedure:

   generate_analog_reports service_name ?time_of_running? ?specified_logfile?

   Where time_of_running is the time in seconds from 1970 (from an ns_time call) of the day you would like the reports to run, and the specified_logfile is the logfile to give to analog, not using the default specified in your .cfg. Note that generate_analog_reports will generate reports for the day_before the time of running. See example.

   Example:
   If I'm running this on December 4, 1998 c. 2:30 pm EST, then an ns_time call returns "912799791". Executing:

   generate_analog_reports foo 912799791

   Will generate the reports for 1998-12-03 from the default logfile specified in foo.cfg.

   Analog will seg fault and dump core if you try to generate reports for a date with no input while specifying a cacheoutfile. The exact command being executed by the server is kept in the reports server's error log. You can search the error log and run the exact command in your shell but you should be sure to:

   1. run it as nsadmin
   2. remove the -q switch (which suppresses warnings and errors
   3. make sure you backslash teh space between +CCACHEOUTFILE and cachefile_name

   
   

6. Client Access:
   Anyone requesting http://reports.arsdigita.com will get hit with a demand to log in. They will then get to /web/reports/index.tcl which will look at ns_user (always the same as the service name) and redirect the user to the appropriate service directory, which will be restricted to that user only. Note that sysadmins with the nsadmin password will be able to go anywhere and the index.tcl script should offer them a dir listing.
   
7. Changes to the Analog Source:
   
   • in init.c:
     • Added the entry:

         {0, "200 OK", 0, 0},

       to the array

       Statuscodes scs[]

   • in input.c:
     • Added the code:

       	  /*Terence hack to match 0 status codes*/
       	  if (*pos == '0')
       	    *pos = '-';
       		

       to the function

       choice parseuint3strict(FILE *f, void *arg, char c) 

   • in process.c:
     • Added the code:

       	  /*terence hack for 0 status codes*/
       	  if (code==0) 
       	    code = 200;
       		

       in the beginning of:

       	void process_data(Filelist *logfilep, FILE *lf, Hashtable **hash,
       		  choice *count, Include **wanthead, Include *ispagehead,
       		  Alias **aliashead, Include *argshead, Include *refargshead,
       		  Dateman *dman, Sizedata *sizes, Tree **tree,
       		  choice *alltrees, choice *lowmem, logical case_insensitive,
       		  char *dirsuffix, unsigned int dirsufflength,
       		  unsigned int granularity)
       		

     • Added the code:

       	    /*terence hack for 0 status codes*.
       	    if (code == 0)
       	      code = 200;
       		

       under the loop

         for (i = 0; scs[i].code < 600; i++)

       in the function

       void codescore(unsigned int code, unsigned long reqs, timecode_t timecode)