[Michael T. Babcock] dnscache log processor

I use dnscache for DNS resolution / caching on all my *nix machines. Its fast, reliable and has a small memory footprint (especially compared to BIND). It also produces very detailed logs of all of its activity.

Wanting to make better use of those logs, I've written dnscacheproc.py, a Python program for analyzing my dnscache logs. At the moment, it processes 'stat' lines and generates statistics about cache motion. If you have the cachestats patch installed, it will also generate a quick hit/miss summary.


Depending on your needs, there are a couple ways you can use this program. You can quickly pipe some dnscache log data through it to get a statistics summary (sample output below) like so: cat /service/dnscache/log/main/current | dnscacheproc.py (of course, change that to match your actual log directory if its different from the installation default).

If you want these statistics on a regular basis, you may wish to use dnscacheproc as a multilog filter. Please read through the multilog documentation and make a backup copy of your /service/dnscache/log/run script before doing anything I describe here though. Breaking your logging can (and will) stop dnscache from working until you fix it.

My log/run script looks like this (note, I've changed the default log output directory):
exec setuidgid logging multilog \
t s102400 n20 '-* cached *' '-* rr *' '-* tx *' /var/log/dnscache \
'+*' s10240 n5 /var/log/dnscache/details \
'-*' '+* stats *' !"/usr/local/bin/dnscacheproc.py" /var/log/dnscache/stats
Doing all this is not necessary to achieve simple logging information, but you may enjoy the benefits; it gives me a basic runtime log of dnscache's activity in /var/log/dnscache/current that doesn't include rr or tx or cached records for brevity. It logs a full detailed version of my logs in /var/log/dnscache/details/current in case I want to take a look at more details when debugging a problem (although rare) and it gives me a statistics-only log in /var/log/dnscache/stats/current which, when processed by the log filter, leaves me statistics output in the @{timestamp}.s files instead of the rotated 'current' data. I can easily check my stats by doing a cat /var/log/dnscache/stats/\@*.s | tail.

Here's a sample of the output:
Overall time: 2.92 seconds
Cache motion: 31260 bytes
Extrapolated motion per day: 923611839 bytes
Including previous data:
Overall time: 16026.71 seconds
Cache motion: 509154 bytes
Extrapolated motion per day: 2744849 bytes
TTL-average 3 day motion: 8234549 bytes
Full-week motion: 19213948 bytes

If you're not familiar with the use of this information, take a quick look at DJB's explanation of calculating an appropriate cache size. As a quick summary, the overall time is the amount of time covered by the log that was just processed. This is based on the TAI64n format timestamps as created by multilog. The cache motion is the number of bytes of cache space that was used in this time period. The extrapolated motion values are how many bytes would be / were used during one day, 3 day and one week time periods. This should give you the information you need to set an appropriate CACHESIZE value to get good hit ratios. I recommend setting your CACHESIZE value slightly higher than the three-day motion value (the next round number perhaps).

Remember to grow your DATALIMIT value by at least as much as your CACHESIZE value before restarting dnscache and that you must actually restart dnscache for these changes to take effect.


Note: You can subscribe to notices of when this project is updated by visiting its freshmeat page.

You will need a copy of my PyTAI library in either the same directory as dnscacheproc.py or your site-packages directory. Yes, I plan to package these both to use disttools at some point.


Stumble it! XFN Friendly Powered by DJBDNS Powered by Zope Valid CSS!

Served by:  Zope 2.7.6

Page Copyright © 2011, Michael T. Babcock. All Rights Reserved.

To contact me, send an E-mail to sawyoursite at this domain.

If you'd really like your mail server reported for spam, send me some junk mail to junk-yum@mikebabcock.me or devnull@mikebabcock.me. This site powered by djbdns