dlog v1.0.0 If you're in a hurry, read INSHORT first - but you may need to come back again :-) Introduction ============ This suite of logfile analyzers examine the logfiles of the following tools: axfrdns, dnscache, qmail, qsmtp (the tcpserver output for qmail-smtpd), rbldns, tinydns and publicfile if they use multilog to store their logs. The suite is broken up into many pieces, and for fully understanding what you should use and when, please continue reading. Usage ===== dlog can be used in three different ways: 1) You only use the data-collecting tools and do the reports and graphing with some other tools, like munin (plugins for munin are provided in this package). 2) You use the data-collecting tools and use dlog to store the data in rrdtool databases. You will create your own graphs and webpages to make use of the data. This requires some scripting on your part - but you can tailor your graphs. 3) You use the data-collecting tools and use dlog to store the data in rrdtool databases. Then you configure different parts of dlog to run at given intervals to provide webpages and graphs. As this README will be concentrating on method 3, you should refer to the SPECS if you choose method 1 or 2, after finishing this document. Requirements ============ Tools: 1) Perl, Lex, Yacc, make & a C compiler for compilation 2) Perl, Lex, Yacc, make, a C compiler for compilation and Perl & RRDtool afterwards 3) Perl, Lex, Yacc, make, a C compiler for compilation and Perl, rrdtool, sort, uniq & head afterwards (sort, uniq and head are only required if you use the list-generating tools). As long as you have a POSIX compliant make, Lex, Yacc and C compiler, dlog should work fine. The sort, uniq and head commands are (as far as I know) part of most UNIX systems. Configuration: The only requirement for your environment is that your logfiles are created by multilog with the "t" parameter - usually the log service for a tool is being started like this: exec setuidgid multilog t ./main Disk space: You need a considerable amount of free space to keep the data that dlog generates. It's difficult to give an estimate, as it depends on how you use dlog, but the rrdtool databases should not take more than 400MB even if use every tool provided in this package. If you use the "list" feature you are completely on your own as it solely depends on the size of the input you give it - but here you need approximately two times the size of the input available to generate the lists you want. Compiling and Installing ======================== $ tar xvzf dlog-1.0.0.tar.gz $ cd dlog-1.0.0 $ perl configure.pl $ make ... $ make install Using dlog ========== There are two different parts in dlog. One simply sums up the activity, and saves the data in rrdtool databases. The other generates lists from your logfiles, and tells you exactly which clients, queries or domains are most active. dodlog.pl can do all the work for you. During the installation, all adjustable parameters should have been set correctly, but they can be altered afterwards. The most important parameters are the location of the RRDTool databases ($rrddatabasebase) and where dodlog should put stat-files and graphs ($statfiles). To make full use of the graphing and reports that dlog provide, you should have the $statfiles directory reachable through a webserver. Enough already.. Examples, please! ./dodlog.pl tinydns init - This will create an rrd database for holding tinydns statistics ./dodlog.pl tinydns update /service/tinydns/log/main/ - This will update the rrd database with information from tinydns logfiles within the last 5 minutes (you can specify timestamps if you need to, but you it asssumes you want the last 5 minutes if you don't specify anything). ./dodlog.pl tinydns graph - This will create graphs for tinydns. Graphs for the last 24 hours, week, month and year are created and stored in $statfiles. ( Instead of tinydns, the following tools can be specified: axfrdns dnscache publicfile qmail qmailqueue qsmtp rbldns. But if you specify qmailqueue you have to specify the path to qmail-qstat instead of the logfile directory, e.g: ./dodlog qmailqueue update /var/qmail/bin/qmail-qstat ) The init command should only be issued once. If you run it again, all the statistics in the tinydns rrd database will be erased. When the data is a year old, RRDTool deletes the information. So don't worry about cleaning up. The update can be executed whenever you want - but internally, the dodlog.pl scripts works in 5 minute intervals. (You can change this but it has great effect on how RRDTool displays your data). The raw tools (everything tool that starts with dlog) works with whatever interval you give at runtime. The graph command can be issued as often as you like; but probably not more often than you run the update command. I run it every time I update the statistics. The tool you specify for dlog reflects which logfiles you are giving, and the names gives themselves except for qsmtpd which is used for analyzing the tcpserver output for qmail-smtpd. There is also this derivative: ./dodlog.pl tinydns raw /service/tinydns/log/mail/ - This will give you the data from the tool directly - the only difference between running the tool (in this case dlogtiny) directly is that here you don't have to specify that you want to analyze the last 5 minutes of data. Please read SPECS to make sure you use the tool optimally - this feature is primarily for integration with other tools. Pfew. A few more tools to go. Another part of dodlog.pl (and dlog in general) is the tools that generates lists of which queries, clients or domains that are most active. Again, examples shows it best: ./dodlog.pl axfrdnsclients list /service/axfrdns/log/main 1036688500 1037293300 This will generate a list of which clients that within the last week, had most frequently requested information from the axfrdns setup. In the above example a text file will be created in the $statfiles directory named axfrdnsclients.1037293300.txt. Similar statistics can be created for other services if you use pass other commands to dodlog.pl: axfrdnsclients dnscacheclients rbldnsclients tinydnsclients Another example: ./dodlog tinydnsqueries list /service/tinydns/log/main 1037206900 1037293300 This will also generate list, but it will contain a list of which queries that within the given period, had been most popular. Similar statistics can be created for other services with the following commands: axfrdnsqueries dnscachequeries tinydnsqueries rbldnsqueries The last three commands to dodlog.pl are: qmailadd qmaildom publicdom They (believe it or not) also create a list. qmailadd creates a list of which email-addresses that are most active on your system - both in (From:) and out (To:). qmaildom creates a list of which domains are most active, and again that goes for both in and out (if you only host one domain, that domain will probably be the most active domain for ingoing mail). publicdom does the same as qmaildom, but for publicfile logfiles. Common for the list-generating tools is that you have to clean up after you made a report. I don't know if you want to save the reports, or throw them out immediately after you have seen them - so that's up to you. Automation ========== After you checked that dlog seems to run fine, you should automate the update and graphing of the statistics. The following entries in your crontab should work for some of you, others will have to alter it a bit: */5 * * * * root /path/to/dodlog.pl tinydns update /service/tinydns/log/main/ */5 * * * * root /path/to/dodlog.pl tinydns graph I make reports about queries, clients and domains once a week: */5 * * * * root /path/to/dodlog.pl tinydnsqueries list /service/tinydns/log/main/ `perl -e 'print time-606900, " ", time;'` That's it ========= I hope that get's you going. If you need further info, please see SPECS.. and check the website: http://dlog.gal.dk/ Links ===== Information about axfrdns, dnscache, qmail, rbldns, tinydns and publicfile can be found on D. J. Bernsteins page http://cr.yp.to. RRDTool is the work of Tobie Oetiker http://www.rrdtool.org qpsmtpd can be found here http://qpsmtpd.develooper.com