Moving Forward

In .Net the defining feature of the libraries is that there is at least 2 different ways you can perform any one task. Each one usually has it’s own subtle uniqueness to it but in the end they all get you from Point A to Point B. The problem is figuring out which approach works best and combining all the different framework elements into a coherent application. In my line of work I write a lot of middlewear. Applications that interfaces with systems made before XML was a blip on the horizon and basically act as glue. One of the most common ways I interact with these applications is by reading their log files and looking for key phrases and elements. I have written more log-watchers than I can count so I’ve finally made the end-all piece of code to watch a log file.

In writing this piece of code you might instantly light up and remember that .Net supports event-driven file watching. There is a very nice IO.FileSystemWatcher class out there that does in fact provide most of the functionality required to implement this design. I used this class for about a week myself. What I came to find is that it is simply riddled with bugs, especially when watching files over the network. The object will simply loose touch with the network folder you are watching. This resulted in a mysterious mandelbug that appears only when mission-critical files need processing. The event-firer will grow eerily quite and nothing will ever be watched again. This is a problem. Upon ripping the FileSystemWatcher from the code and replacing it with a simple thread-based polling piece of code the problem was solved. The application that runs this code is now one of the most stable in the building, with an uptime approaching 2 years and counting.

So without further discussion here is my code for a simple watcher class that will watch either a directory with rolling log files or a single file.

WatcherPhrase.vb

Public Class WatcherPhrase
    Private _name As String
    Private _regex As Regex

    Public ReadOnly Property Name As String
        Get
            Return _name
        End Get
    End Property

    Public ReadOnly Property Matcher As Regex
        Get
            Return _regex
        End Get
    End Property

    Public Sub New(ByVal Matcher As Regex, ByVal Name As String)
        _name = Name
        _regex = Matcher
    End Sub
End Class

FileWatcherEventArgs.vb

Public Class FileWatcherEventArgs
    Inherits EventArgs

    Private _matcher As WatcherPhrase
    Private _file As IO.FileInfo
    Private _match As String

    Public ReadOnly Property matcher As WatcherPhrase
        Get
            Return _matcher
        End Get
    End Property
    Public ReadOnly Property file As IO.FileInfo
        Get
            Return _file
        End Get
    End Property
    Public ReadOnly Property match As String
        Get
            Return _match
        End Get
    End Property

    Public Sub New(ByVal matcher As WatcherPhrase, ByVal file As System.IO.FileInfo, ByVal match As String)
        _matcher = matcher
        _file = file
        _match = match
    End Sub
End Class

FileWatcher.vb

Public Class FileWatcher
    Public Enum inputTypes
        File
        Folder
    End Enum

    Private _inputPath As String
    Private _inputType As inputTypes

    Private _isRunning As Boolean = False
    Private _stoping As Boolean = False
    Private _workerThread As System.Threading.Thread
    Private _matcherList As New List(Of WatcherPhrase)

    Public Event matchFound(ByVal sender As Object, ByVal e As FileWatcherEventArgs)

    Public Sub New(ByVal inputPath As String, ByVal inputType As inputTypes)
        _inputPath = inputPath
        _inputType = inputType
    End Sub

    Public Sub New(ByVal inputPath As String, ByVal inputType As inputTypes, ByVal matcher As WatcherPhrase)
        _inputPath = inputPath
        _inputType = inputType
        _matcherList.Add(matcher)
    End Sub

    Public ReadOnly Property isRunning As Boolean
        Get
            Return _isRunning
        End Get
    End Property

    Public Property matcherList As List(Of WatcherPhrase)
        Get
            Return _matcherList
        End Get
        Set(ByVal value As List(Of WatcherPhrase))
            _matcherList = value
        End Set
    End Property

    Public Sub Start()
        _workerThread = New System.Threading.Thread(AddressOf Process)
        _workerThread.Start()
        _isRunning = True
    End Sub

    Public Sub StopMe()
        _stoping = True
        If _workerThread IsNot Nothing AndAlso _workerThread.IsAlive = True Then
            _workerThread.Join()
        End If
        _isRunning = False
    End Sub

    Private Sub Process()
        Dim lastFile As IO.FileInfo
        Dim lastPosition As Long
        Dim currentFile As IO.FileInfo

        If _inputType = inputTypes.File Then
            currentFile = New IO.FileInfo(_inputPath)
        End If

        While True
            If _stoping = True Then
                Return
            End If

            ' Main loop
            If _inputType = inputTypes.Folder Then
                Dim dirSearch As New IO.DirectoryInfo(_inputPath)
                Dim mostModifiedFile As IO.FileInfo = (From c In dirSearch.GetFiles Order By c.LastWriteTime Descending).FirstOrDefault
                If lastFile Is Nothing Then
                    lastFile = mostModifiedFile
                    lastPosition = mostModifiedFile.Length - 1
                End If

                If currentFile IsNot Nothing Then
                    ' If the last modified file has changed we assume a new logfile has been created.
                    If lastFile.FullName <> mostModifiedFile.FullName Then
                        lastFile = mostModifiedFile
                        lastPosition = 0
                    End If
                End If

                currentFile = mostModifiedFile
            End If

            Try
                If currentFile IsNot Nothing AndAlso lastPosition < currentFile.Length Then
                    Dim oFile As New IO.FileStream(currentFile.FullName, IO.FileMode.Open, IO.FileAccess.Read, IO.FileShare.ReadWrite)
                    oFile.Seek(lastPosition, IO.SeekOrigin.Begin)
                    Dim oReadStream As New IO.StreamReader(oFile)
                    Dim str As String = oReadStream.ReadToEnd

                    If str.Contains(vbCrLf) = True Then
                        Dim list As New List(Of String)(str.Substring(0, str.LastIndexOf(vbCrLf)).Split(New Char() {vbCrLf}))
                        For Each oEl As String In list
                            For Each oReg As WatcherPhrase In _matcherList
                                Dim matchResult = oReg.Matcher.Match(oEl)
                                If matchResult.Success = True Then
                                    RaiseEvent matchFound(Me, New FileWatcherEventArgs(oReg, currentFile, oEl))
                                End If
                            Next
                        Next
                        lastPosition = lastPosition + str.LastIndexOf(vbCrLf) + 1
                        lastFile = currentFile
                    End If
                End If
            Catch ex As Exception

            End Try
        End While
    End Sub
End Class

And how does it all fit together? Here is a very simple example of the
classes’ usage.

FileWatcherTest.vb

Public Class Tester
    Private WithEvents _fileWatcher As FileWatcher
    Public Sub StartTest()
        Dim oMatchPhrase As New WatcherPhrase(New System.Text.RegularExpressions.Regex("An error has occured."), "Error Log Pattern Matcher")
        _fileWatcher = New FileWatcher("C:\logs", FileWatcher.inputTypes.Folder, oMatchPhrase)
        _fileWatcher.Start()
    End Sub

    Private Sub HandleFileWatcher(ByVal sender As Object, ByVal e As FileWatcherEventArgs) Handles _fileWatcher.matchFound
        Debug.WriteLine("Match Found: " & e.matcher.Name & " found a match in " & e.file.Name & " at """ & e.match & """")
    End Sub

    Public Sub EndTest()
        _fileWatcher.StopMe()
    End Sub
End Class

And there you go, one fully functional log watcher with each and every
bug worked out. The following code is in use every day with hundreds
of log files over many different testing expressions!