Comments on: Feedback and Boredom Result in 35% Performance Boost for Loghetti http://www.protocolostomy.com/2008/03/14/feedback-and-boredom-result-in-35-performance-boost-for-loghetti/ Made with only the finest 1's and 0's Wed, 10 Mar 2010 20:02:17 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Kent Johnson http://www.protocolostomy.com/2008/03/14/feedback-and-boredom-result-in-35-performance-boost-for-loghetti/comment-page-1/#comment-193 Kent Johnson Mon, 17 Mar 2008 02:02:13 +0000 http://www.protocolostomy.com/2008/03/14/feedback-and-boredom-result-in-35-performance-boost-for-loghetti/#comment-193 A smarter regex makes a huge improvement. The current regex has to do a lot of backtracking because of all the .* matches. Try this one: re.compile(r'(\d+\.\d+\.\d+\.\d+) ([^ ]*) ([^ ]*) \[([^ ]*) [^\]]*\] "([^"]*)" (\d+) ([^ ]*) "([^"]*)" "([^"]*)"') I have a few smaller improvements, too. Drop me an email and I will send them to you, it is a bit much for a comment. A smarter regex makes a huge improvement. The current regex has to do a lot of backtracking because of all the .* matches. Try this one:
re.compile(r’(\d+\.\d+\.\d+\.\d+) ([^ ]*) ([^ ]*) \[([^ ]*) [^\]]*\] “([^"]*)” (\d+) ([^ ]*) “([^"]*)” “([^"]*)”‘)

I have a few smaller improvements, too. Drop me an email and I will send them to you, it is a bit much for a comment.

]]> By: m0j0 http://www.protocolostomy.com/2008/03/14/feedback-and-boredom-result-in-35-performance-boost-for-loghetti/comment-page-1/#comment-197 m0j0 Sat, 15 Mar 2008 14:17:12 +0000 http://www.protocolostomy.com/2008/03/14/feedback-and-boredom-result-in-35-performance-boost-for-loghetti/#comment-197 Doug -- Thanks - that's a great article, as it talks about these optimizations in the context of *exactly* what I'm doing. Thanks again! Kent -- I haven't posted the revised code. I was hoping that the apachelogs author would make his project public so I could just pass on improvements and have them accepted or not instead of maintaining a separate version distributed with loghetti. In the meantime, I was hoping to pull most of my custom code *out* of that module so that there aren't duplicate modules flying around. I'll just put it somewhere in the Filter class, which is what I started doing to implement the 'lazy' features. Thanks for the tips above - they mirror some thoughts I'm having after reading the article linked by Doug. There's lots of great stuff in there. Gimme another week, and I'll post the revised code. Doug — Thanks – that’s a great article, as it talks about these optimizations in the context of *exactly* what I’m doing. Thanks again!

Kent — I haven’t posted the revised code. I was hoping that the apachelogs author would make his project public so I could just pass on improvements and have them accepted or not instead of maintaining a separate version distributed with loghetti. In the meantime, I was hoping to pull most of my custom code *out* of that module so that there aren’t duplicate modules flying around. I’ll just put it somewhere in the Filter class, which is what I started doing to implement the ‘lazy’ features.

Thanks for the tips above – they mirror some thoughts I’m having after reading the article linked by Doug. There’s lots of great stuff in there. Gimme another week, and I’ll post the revised code.

]]> By: Kent Johnson http://www.protocolostomy.com/2008/03/14/feedback-and-boredom-result-in-35-performance-boost-for-loghetti/comment-page-1/#comment-196 Kent Johnson Sat, 15 Mar 2008 13:28:46 +0000 http://www.protocolostomy.com/2008/03/14/feedback-and-boredom-result-in-35-performance-boost-for-loghetti/#comment-196 Did you post the revised code? In _ApacheLogFileGenerator.Generator: - hoist the lookup of self.r.match out of the loop - Create the ApacheLogLine with log_line = ApacheLogLine(*m.groups()) to cut out a bunch of method calls. BTW you could simplify this code by making Generator be ApacheLogFile.__iter__; the nested class and delegation from __iter__ are not needed. BTW for your grep test did you use the same regex or did you just grep 404? Did you post the revised code?

In _ApacheLogFileGenerator.Generator:
- hoist the lookup of self.r.match out of the loop
- Create the ApacheLogLine with
log_line = ApacheLogLine(*m.groups())
to cut out a bunch of method calls.

BTW you could simplify this code by making Generator be ApacheLogFile.__iter__; the nested class and delegation from __iter__ are not needed.

BTW for your grep test did you use the same regex or did you just grep 404?

]]> By: Doug Hellmann http://www.protocolostomy.com/2008/03/14/feedback-and-boredom-result-in-35-performance-boost-for-loghetti/comment-page-1/#comment-195 Doug Hellmann Sat, 15 Mar 2008 11:57:08 +0000 http://www.protocolostomy.com/2008/03/14/feedback-and-boredom-result-in-35-performance-boost-for-loghetti/#comment-195 This article should offer some other tips for boosting performance: http://effbot.org/zone/wide-finder.htm This article should offer some other tips for boosting performance: http://effbot.org/zone/wide-finder.htm

]]> By: Toneby http://www.protocolostomy.com/2008/03/14/feedback-and-boredom-result-in-35-performance-boost-for-loghetti/comment-page-1/#comment-194 Toneby Sat, 15 Mar 2008 07:47:14 +0000 http://www.protocolostomy.com/2008/03/14/feedback-and-boredom-result-in-35-performance-boost-for-loghetti/#comment-194 "grep | wc -l" isn't the optimal way, use "grep -c" instead and you should get even shorter times since there are no extra processes and pipes involved. Great you are getting you log checker more optimized, I know it sucks waiting around to get the results back from investigation large logfiles. I fairly often have to do it with multi gigabyte files. “grep | wc -l” isn’t the optimal way, use “grep -c” instead and you should get even shorter times since there are no extra processes and pipes involved. Great you are getting you log checker more optimized, I know it sucks waiting around to get the results back from investigation large logfiles. I fairly often have to do it with multi gigabyte files.

]]>