EMD part1 – logfunneld

From a requirements perspective, we have a couple of long standing requirements.  These are:

• Scalable
• No down time
• Fault tolerant

From a functional perspective, we need something that:

• Takes in events
• Parses, processes, and routes these events
• Stores events
• Displays events
• Logs, stores, and archives all events.

This is good enough to get things rolling.

————-

From a simple process perspective, we need to start out with something simple that reads in one or more log files, processes that data, and routes it accordingly.  In effect, its a probe of sorts.

So, check out the following link :

http://poe.perl.org/?POE_Cookbook/Watching_Logs

We can start by opening a single log file, parsing the input, and going from there.  In the cookbook example, we see a program that opens one or more log files and processes them like a tail -f <filename> for each filename.  As input lines are realized, an event is yielded to the got_line() subroutine.  This sub only prints the $_[ARG0] data and increments $_[HEAP]->{first} which is a counter in the Heap keyed by the word first.

So, to start off our logfunneld process:

=================

#!/usr/bin/perl

# logfunneld process

use warnings;
use strict;
use POE qw( Wheel::FollowTail );
# Here are the log files to watch.
# lets stick with /var/adm/messages first and foremost
my %logs_to_watch = (
#  cron  => "/var/log/cron",
#  mail  => "/var/log/maillog",
#  ppp   => "/var/log/ppp.log",
#  httpd => "/var/log/httpd-access.log",
  msg   => "/var/log/messages",
);
POE::Session->create(
  inline_states => {
    _start => \&begin_watchers,
# This kicks off the session with starting the watchers…
    got_record   => \&got_record,
    # Handle log resets and errors the same way for each file.
    log_reset => \&generic_log_reset,
    log_error => \&generic_log_error,
  }
);
# Now for the tricky stuff...
sub begin_watchers {
  my $heap = $_[HEAP];
  while (my ($service, $log_file) = each %logs_to_watch) {
    my $log_watcher = POE::Wheel::FollowTail->new(
      Filename   => $log_file,
      InputEvent => $service . "_record",
      ResetEvent => "log_reset",
      ErrorEvent => "log_error",
    );
    $heap->{services}->{$log_watcher->ID} = $service;
    $heap->{watchers}->{$log_watcher->ID} = $log_watcher;
  }
}
# Handle log resets the same way for each file.  Simply recognize that
# the log file was reset.
sub generic_log_reset {
  my ($heap, $wheel_id) = @_[HEAP, ARG0];
  my $service = $heap->{services}->{$wheel_id};
  print "--- $service log reset at ", scalar(gmtime), " GMT\n";
}

# Handle log errors the same way for each file.  Recognize that an
# error occurred while watching the file, and shut the watcher down.
# If this were a real log watcher, it would periodically try to resume
# watching the log file.
sub generic_log_error {
  my ($heap, $operation, $errno, $error_string, $wheel_id) =
    @_[HEAP, ARG0, ARG1, ARG2, ARG3];
  my $service = $heap->{services}->{$wheel_id};
  print "--- $service log $operation error $errno: $error_string\n";
  print "--- Shutting down $service log watcher.\n";
  delete $heap->{services}->{$wheel_id};
  delete $heap->{watchers}->{$wheel_id};
}
sub got_record {
    my $record = $_[ARG0];
    print "$record\n";
}
# Run the watchers.  The run() method will return after all the
# watchers end.
$poe_kernel->run();
=====================
What we have here is a process that will watch one or more log files and pint to stdout every line.  Pretty simple huh.
The interesting thing about POE is that it organizes your work into subs that are called or yielded via callbacks.
This will literally process hundreds of lines per second and sleep process wise, most of the time.  Its not waiting to see a line via wait or sleep, nor would we want to.  These are BLOCKING CALLS which halt your program from doing anything else until this sleep or wait is done.  In effect, blocking code is bad for our first requirement - Scalability.
This is the basic premise behind logfunneld.  How simple is that?
Share this:
Email
Print
Facebook
Twitter
Tumblr
LinkedIn
Like this:
Like Loading...


	
Related