Analog 5.22: Form interface and CGI program

Important: For security reasons, you must not attempt to run analog itself as a CGI program, or even leave it in the directory or folder with your web files or CGI programs. When the form interface runs analog for you, it checks that analog isn't given any dangerous options. Without this check, your system could be vulnerable to attack.

Please don't try and set up the form until analog has been set up and is running properly on its own. It just adds another level of complexity to troubleshoot. And unlike analog itself, the form interface will not run "out of the box". You have to read this section to find out how to set it up.

The form interface is suitable for ordinary users to use, but it needs to be set up by a system administrator or other expert. In order to set it up, you have to be running a web server. You need to know what CGI programs are, where they live on your server, and how to set up their permissions properly. You also need to know how to write HTML forms. I shall assume this level of background knowledge for the rest of this section. And you have to be running Perl 5.001 or later: see Technical details below for other system requirements. (Actually, if you're on Windows and don't have Perl, you can download an executable version of the form interface from the helper applications page.)

Warning: CGI programs can contain security loopholes which allow an unscrupulous user to harm your system. (If you don't know about this, you shouldn't be running CGI programs at all. Read and understand the World Wide Web Security FAQ and the CGI Security FAQ first.) I have tried to make this form interface safe, but I cannot guarantee it. Even the most carefully-designed CGI programs can accidentally have serious security bugs. And I take no responsibility if anything goes wrong: you use it at your own risk. (See the licence.) Furthermore, you should be aware that unless you take special measures like password protection or limiting anlgform.pl to specific hostnames, setting up the form interface implies making analog executable, and your logfiles analysable, by anyone on the internet. There are more notes on security design in this program towards the end of this section.

The form interface consists of two parts: a form (called anlgform.html) to choose the options, and a cgi program (called anlgform.pl) to pass them to the analog program. Both anlgform.html and anlgform.pl must be configured to your system before they will work at all. There are instructions at the top of both files explaining how to do this.

The form which is distributed with the program should only be regarded as an example form. You can find forms in languages other than English in the lang directory. Or you can write your own if you prefer. In fact you don't actually need the form at all: if you want just to create a link to the cgi program, with the arguments passed after a question mark in the URL in the usual way, then that's fine.

Logfile name: <input type=text name="LOGFILE">

<select name=LOGFILE size=1>
  <option value="/var/log/apache/fred"> Fred's logfile
  <option value="/var/log/apache/jane"> Jane's logfile
</select>

There are a few commands which you can't specify on the form for security or performance reasons. The full list is *LOGFORMAT, LANGFILE, DESCFILE, HEADERFILE, FOOTERFILE, UNCOMPRESS, OUTFILE, CACHEOUTFILE, ERRFILE, LOCALCHARTDIR, DNS and SETTINGS; and the person setting up the form can add more. See the security notes below for the reasons for these exclusions, and for some more commands you might want to add to the forbidden list. You can, if you prefer, specify the commands which are allowed, rather than those which are forbidden.

Alias this file: <input type=text name="FILEALIAS1">
To this one: <input type=text name="FILEALIAS2">

Then there are FLOOR commands. To avoid users of the form having to know the syntax of these commands, you can if you want specify them in two halves, FLOORA and FLOORB, and they will be stuck together. For example, the form distributed with the program specifies

<br>Include all domains with at least
<input type=TEXT name="DOMFLOORA" maxlength=6 size=6>
<select name="DOMFLOORB">
  <option value=r>requests
  <option value=p>requests for pages
  <option value=b selected>bytes
</select>

Secondly, you can specify other configuration files to be included at specific times. When analog is called by the CGI program, it first processes the default configuration file as usual. Then it processes any configuration file specified by an option with name cg. Then it processes all the other commands which the CGI program specifies. After that, it processes any configuration file specified by an option with name cm. Finally, it processes the mandatory configuration file as usual. (You may therefore want two copies of analog, one for form use and one for non-form use, with different configuration files compiled in.) Note that the commands in the default and mandatory configuration files will contribute to the configuration: some of them may even override options specified on the form. For example, if the default configuration file contains an INCLUDE command, this may cause INCLUDE and EXCLUDE commands specified on the form to behave unexpectedly.

There are a couple of commands which the form always sets. These may override what you have set elsewhere. First, it sets either DNS READ (if a DNSFILE is set on the form) or DNS NONE (otherwise). You can override this behaviour in the mandatory configuration file, but you are likely to run into timeout problems if you do. Secondly, it always sets WARNINGS FL, so that the less important warnings don't fill up your server's error log. You can override this by sending an explicit WARNINGS command from the form.

You won't get pie charts on the form unless you set a CHARTDIR and LOCALCHARTDIR in your default configuration file (LOCALCHARTDIR is disabled from the form for security reasons). And even if you do this, there will be a problem if two users try and run the form interface at the same time, because they will be trying to write the same images, so they may see broken images or each other's charts.

There is one small point about compressed logfiles. For security reasons, when using the form interface you need to specify the full pathname to the uncompression command in the UNCOMPRESS command in your configuration file.

Again for security reasons, analog checks the input from configuration commands more carefully when using the form interface before outputting it. One side-effect of this is that the JAPANESE-JIS character set won't work. Use one of the other Japanese character sets instead.

Form troubleshooting

First, does analog run properly on its own without anlgform?

Next, you can run anlgform.pl from the (DOS or Unix) command line. This is good enough to debug most problems. You can specify options in pairs like this:

anlgform.pl qv=1 LOGFILE=/some/log REQINCLUDE=pages

If it still doesn't work, check the following points:

1. Have you edited anlgform.pl and anlgform.html as instructed at the top of those files?
2. Do other CGI programs work on your server? Is anlgform.pl in the right place to be recognised as a CGI program by the server?
3. Look in the server's error log for clues. You might want to set WARNINGS ON before you do this, because by default only warnings in categories F and L are reported.
4. Sometimes it's helpful to set the ERRFILE in your analog configuration file (it won't work from the form) to catch any errors and warnings which may be getting lost. This is especially true on IIS which incorrectly sends errors to the browser instead of to an error log. If you are using Internet Explorer you will probably also need to disable the "friendly" error messages so that you can see the actual error message.
5. Are all relevant files (analog itself, logfiles, configuration files, auxiliary files such as domain files...) executable/readable by your web server?
6. If some form options don't seem to take effect, then check whether they are being overridden by a command in a configuration file. (Although SETTINGS is forbidden from the form, you might find it useful to set it in your default configuration file.)
7. If you get a long wait, then no data returned, the server is probably timing out the request before analog has finished. The remedy is to increase the timeout interval.
8. As explained above, the form always sets DNS READ or DNS NONE, and WARNINGS FL, overriding your default configuration file.
9. Again as explained above, pie charts may not appear or may appear wrongly.
10. Again as explained above, uncompressing of compressed logfiles doesn't work unless you use the full pathname in the UNCOMPRESS command.
11. And once again as explained above, the JAPANESE-JIS character set won't work from the form interface.

Security notes

First, you should think about who can run the form interface. Unless you take special measures like password protection or limiting anlgform.pl to specific hostnames, adding the form interface to your site implies making analog executable, and your logfiles analysable, by anyone on the internet. There are obvious concerns both about privacy and about the load on your system.

Certain commands are ignored by anlgform.pl and not passed to analog. The list of them can be found at the top of anlgform.pl. Here are the reasons for them. HEADERFILE and FOOTERFILE would place any file on your system within the output. The *LOGFORMAT commands would also allow any file to be read, because someone could designate each line to be a single filename and then just list the filenames. OUTFILE, CACHEOUTFILE, ERRFILE and LOCALCHARTDIR would allow people to write to your filespace; ERRFILE would also divert warnings away from your server's error log. UNCOMPRESS would allow a user to execute any command. DNS is forbidden because setting it higher than READ would normally cause the process to time out. CGI would allow the user to generate syntactically incorrect output.

None of the above should be deleted (unless you are really, really sure that it's completely impossible for anyone other than yourself to run anlgform.pl). There are three other commands which are forbidden by default but which you could consider removing from the forbidden list. SETTINGS is included because it will give away the locations of some files on your system. But it is useful for diagnostic purposes, and you could consider removing it temporarily if you have trouble setting up the form. The other commands which are included are LANGFILE and DESCFILE. They are included because it is possible that another file could be exactly the right number of lines long to be accepted as a language file or report descriptions file, and then parts of it would get into the output. But it would have to be exactly the right number of lines long first. These commands shouldn't really be needed if your copy of analog is installed correctly, because the LANGUAGE command should find the right files. But if you want them, and you're prepared to take the risk described above, you can remove LANGFILE and/or DESCFILE from the list.

There are other commands which you might consider adding to the list. For example, it is theoretically possible (though rather unlikely), that another file on your system could conform sufficiently closely to one of the predefined log formats that analog could be persuaded to analyse it and so reveal some of its contents. If you're worried about this, or even if you want to force only one particular logfile to be analysed from the form, you can add the LOGFILE command to the list of forbidden commands. And you could add DOMAINSFILE for similar reasons. Or if you wanted to stop a user having control over which analog warnings were written to the error log, you could add WARNINGS to the list. (Possible attempted security violations detected by anlgform will always be written.)

You can of course add any command you like to the list. For example, a user can use any configuration file on your system unless you add CONFIGFILE. If you add a command, you must also add any aliases for it. Have a look in the source file globals.c for the same command under different names -- some commands have legacy names which I don't admit to in the documentation.

For more certainty, you can, if you prefer, configure anlgform so that you specify the commands which are allowed, rather than those which are forbidden. See the top of anlgform.pl for how to do this.

The arguments to LOGFILE and CACHEFILE commands are checked for containing only certain allowed characters (specifically, letters, digits, /\.:_*? space, and - between two {letter, digit, underscore}'s). This is because they could match an UNCOMPRESS command and thus be passed to the shell when the uncompress command is popen()'ed.

Apart from that, command names are checked for containing only letters and the digits 1 and 2; and the arguments to commands are checked for not containing control characters (actually characters 0-32 and 127-159; in particular newline characters are prohibited). The length of the commands isn't checked by anlgform.pl, but buffer overflow shouldn't be an issue as configuration commands are checked for length by analog.

By the way, the reason that I advise that analog itself shouldn't be used as a CGI program is that some servers, notably Microsoft IIS, allow users to pass command line arguments into a CGI program. And even if the program doesn't return the proper CGI headers, the output can be sent back to the user. This means that all the above checking of arguments is then thwarted. Of course, on servers on which you can't pass command line arguments to a CGI program, there are not the same security concerns, but then analog isn't very useful as a CGI program because if you can't pass any arguments, you can only get the default output.

Technical details

On Windows, you have to associate the .pl extension with the Perl executable so that Perl scripts are executed by Perl.

anlgform.pl will understand the GET or POST methods of form submission. The HTML spec says that GET should be used when, as in this case, running the program has no side effects. However, section 15.1.3 of the HTTP spec says that POST should be used if some of the options being passed might be confidential. Also, very long URLs, formed by specifying lots of options, can cause trouble to some older servers. So anlgform.html uses the POST method by default. However, the GET method will also work. For example, you could make a normal link to anlgform.pl with options specified after a question mark in the usual GET way.

Need help with analog? Use the analog-help mailing list.

[ Top | Up | Prev | Next | Map | Index ]