Download Access Logs with Perl - Parsing V
(60) Mar 14th 2008 04:38AM 'Developer' 'perl'
This post describes how to download files using Perl and FTP.
***
An earlier post, Log Parsing III provided steps to scan web access logs in Perl and provided a sample script which scan's a bunch of logs and then creates a web page of links to Google searches. Yet another Log Parsing Post, goes a step further and creates an iPhone formatted page of Google searches. Parsing for Google searches is just one of many, many things that can be done using access logs.
***
Since my ISP is Yahoo and I'm not paying for their premium account, I don't have a Unix shell to run perl scripts. Before scanning a log, it needs to be copied from Yahoo to my personal PC. FTP is used here.
***
Perl makes it easy to copy, or download, files using FTP.
***
To use FTP in Perl, first make sure the package Net::FTP is referenced in the script, usually at the top of the script in a use statement. The Net::FTP package lets you create simple FTP clients which can login, change directorys, set transfer modes, download or upload files, and logout.
***
Below, the getFile() function wraps Net::FTP method calls into a subroutine that downloads access log files. You provide the user ID, password, hostname, and other parameters. The function logs into the remote machine, changes to the target folder, and downloads files with names beginning with "access". It returns a list of downloaded filenames. A regex skips files with names that don't begin with "access".
#-----------------------------------------------------------------
# Function: getFile
# Purpose: Use ftp to copy log files from a remote machine to
# the local machine.
#-----------------------------------------------------------------
sub getFile
{
my $filename = shift; # local filename (full path)
my $from_host = shift; # To Host
my $from_folder = shift; # To Folder
my $uid = shift; # ftp login id
my $pwd = shift; # ftp login password
my ($ftp);
my ($currentDir, $result);
$ftp = Net::FTP->new($from_host) or die "Can't connect: $@\n";
#--- Login
$ftp->login($uid, $pwd ) or die "Can't login with $uid and $pwd\n";
#--- get name of current directory
$currentDir = $ftp->pwd();
#--- Change to the target directory
$ftp->cwd("$from_folder") or die "Can't change directory\n";
#--- get name of current directory on remote machine
$currentDir = $ftp->pwd();
$ftp->ascii(); # set ascii transfer mode
#--- print files in remote Dir
my @remotefiles;
my @localfiles;
undef @localfiles;
@remotefiles = $ftp->dir();
foreach my $longfilename (@remotefiles) {
if ($longfilename =~ /access\w*\.log/) {
$filename = $&;
$filename =~ s/^\s+//g;
print "\nGetting filename = [$filename] ";
push(@localfiles, $filename);
}
$ftp->get($filename);
}
#--- Disconnect
$ftp->quit() or warn "Couldn't quit. Oh well.\n";
return @localfiles;
}
This function could be modified to move any file or combination of files.
|