With the addition of a little bit of elbow grease infrastructure mode on the HTC EVO is officially working. Over the coming days I hope to integrate it with android-wifi-tether and release the first and only application for any Android platform to support infrastructure mode tethering that doesn't cost $30 a month. This application serves as literally a step-in replacement for the Sprint Hotspot application with the complete functionality set, minus the cost. Support for WEP and WPA comes standard out of the box, as well as open functionality. Unlike the Sprint application this method will not impose an artificial limit on the number of users to connect to your hotspot. 

How does it work?

While I'm not ready to release the source code, mostly due to the fact the small modifications I made will leak memory like a sieve, I will give you the general overview behind the hack. The Broadcom drivers expose some additional functionality that is related to the wireless access point mode of the device. Using a system call that has absolutely no documentation and the driver from the bravo kernel I was able to build up a call in user-space C that emulates the structures expected by the hidden Android system call. 

Really that's all there is to it, throw some timing into the mix and just the right amount of luck and you have yourself a working wireless driver. From the kernel log:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

<4>[19452.040679] penguin, get AP_PROFILE_SET
<4>[19452.041076] wl_iw: set ap profile:
<4>[19452.041442]       ssid = AndrewsAndroidAP
<4>[19452.042144]       security = wpa-psk
<4>[19452.042510]       key = 1234567890
<4>[19452.042877]       channel = 0
<4>[19452.043243]       max scb = 2
<4>[19454.110931] Set auto channel = 1
<4>[19454.112182] wl_iw_setap: do passhash...
<4>[19454.210723] [00]: aa2f2f2c
<4>[19454.210845] [01]: f4081ab1
<4>[19454.211059] [02]: ccc1d613
<4>[19454.211181] [03]: 887fd525
<4>[19454.211273] [04]: b51d0c01
<4>[19454.211395] [05]: 781b89b1
<4>[19454.211608] [06]: b5de9c57
<4>[19454.211730] [07]: 2f8812e2
<4>[19454.211853] wl_iw_setap: passphase = 2c2f2faab11a08f413d6c1cc25d57f88010c1db5b1891b78579cdeb5e212882f
<4>[19454.222106] ap setup done
<4>[19454.224487] send AP_UP

Permalink | Leave a comment  »

As I read through the internet my number one complaint when it comes to all the wonderful work done by developers is the lack of documentation of the process taken to get to the end result. While I do love the fruits of their labor I'm so interested in the processes that led them there and what they learned along the way. 

With that in mind, after recently working to help in a small way to fixing the wonderful android-wifi-tether application I'd like to share what I found along the way and the steps I took to get there.

Assessing the situation

The first thing I did was to troubleshoot what exactly the issue was. From the already existing bug reports I learned that there was some serious issues with communication and that they were solved by running a ping from the device to the laptop connecting. This immediately set me off, pings by themselves do not have magical characteristics so I fired up WireShark to troubleshoot. WireShark is a network protocol analyzer and allows you to monitor TCP/IP traffic going to and from your machine (and in some cases other machines).

In addition to this information I identified the chipset of the wireless driver, the bcm4329, and realized that many other phones had the same chipset. The HTC Desire was having similar problems so this obviously wasn't an isolated incident, however the Nexus One has the same chip but no problems. Because all these phones share the same driver much information was available. 

Finally I took a look at the Sprint hotspot utility included with the phone. It wasn't having an ounce of trouble so this obviously was possible and implemented on this phone. I captured some kernel debugs and started hacking away at it.

The Art of Problem Solving

Next I had to bring all this information together and figure out what the heck was going wrong. I will let you in on the develop's secret weapon at this point: their notes. It is absolutely nessessary to keep good notes while debugging a large, complex problem like this one, else you will end up chasing your tail. To do this I use a combination of pen-and-paper and Microsoft OneNote. 

Collect absolutely everything you figure out and refer to it often. This will save you much time and make life easier when you go to post on a newsgroup with your findings.

The Actual Problem

So what was the actual problem? After much chasing it boiled down to poor WiFi drivers. When the kernel module is loaded an optional parameter, firmware_path, is passed to it. The kernel will grab the firmware file at this path and load it into the broadcom chip for operation. This was done to ensure that firmware bugs can be updated out of the driver in future releases and to support the exact kind of dynamic loading behavior that we were looking for.

After installing tcpdump on the Android device and doing a capture I realized that the packets were not in fact making it to the device. Watching the traffic my laptop would repeatedly send ARP requests to the broadcast address but the phone would never show them, not even in the tcpdump. It wasn't that it was ignoring the ARP requests, it was filtering them on a very low level. Why this was happening is a mystery to me at the moment, although I have a feeling it has to do with power-management technology.

Here's the summary:

1. Laptop sends broadcast packet to device (ARP or DHCP request)
2. Device completely ignores laptop.
3. Change laptop to static IP address, start constant ping on device to device
4. Device ignores laptop 99% of the time

Obviously something was just blocking broadcast TCP packets from being received on the device. This to me seems like power saving technology hard at work.

That sucks, what a dead-end. The key came when I looked at the different firmware files sitting in the /etc/firmware directory. For whatever reason, HTC has included one labeled "fm_bcm4329_ap.bin". Now, I don't know where you come from, but in my world that stands for access point. 

There was much finger-crossing as we loaded the _ap.bin driver into the kernel and tested android-wifi-tether. Lo and behold, it worked! In my mind we honestly got really lucky, the patch to the application was written in 30 seconds and the issue was grossly superficial. If HTC was trying to keep us from unlocking this feature they certainly didn't try very hard.

Next Stop: Infrastructure Mode? 

The buzz has been that 2.2-Froyo is going to include built-in tethering support, and the world will rejoice with happiness as carriers eliminate their lucrative data plans. I don't see much changing, the carriers won't have to write their own WiFi apps but I'm certain they will want to monetize the feature. If what AT&T just pulled shows us anything it is that the carriers are not out to help us, they are out to nickel and dime us to death and they will continue to do so until someone comes along with a better business model. 

I think there will be a continuing roll for hotspot applications in the future so I believe that taking the time to create code necessary for infrastructure mode sounds like a worthy investment. While examining the bcm4339.ko driver source code in the bravo kernel I've found some pretty interesting system calls surrounding SOFTAP mode as it is called. They are easy to trace by logging the kernel messages. The sprint hotspot application takes a completely different approach to initializing:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

Sprint Initialization:
<4>[ 5350.746154] Dongle Host Driver, version 4.218.161.0
<7>[ 5351.397888] set brightness = 164
<3>[ 5353.265350] mmc0: Command timeout (cmd= 8 mci_st=00000004)
<7>[ 5353.289428] mmc0: queuing CIS tuple 0x91 length 3
<6>[ 5353.289794] mmc0: new high speed SDIO card at address 0001
<4>[ 5353.295410] DHD: dongle ram size is set to 294912(orig 294912)
<4>[ 5353.395233] wifi: Disable 802.11n
<4>[ 5353.494415] Enter set packet filter
<4>[ 5353.496948] start to add pkt filter 100
<4>[ 5353.503875] Enter set packet filter
<4>[ 5353.506530] start to add pkt filter 102

<4>[ 5353.512969] dhd_preinit_ioctls:str=0x00112233445538e7d819d70c08004500000000000000000000000000000000000

<6>[ 5353.514648] eth0 (): not using net_device_ops yet
<4>[ 5353.518615] eth0: Broadcom Dongle Host Driver mac=38:e7:d8:19:d7:0c


<6>[ 5354.037048] led_hotspot_status_value = 1
<4>[ 5354.129608] penguin, get AP_PROFILE_SET
<4>[ 5354.131866] wl_iw: set ap profile:
<4>[ 5354.133941]       ssid = EVO ******
<4>[ 5354.137176]       security = wpa-psk
<4>[ 5354.137573]       key = 1234567890
<4>[ 5354.152282]       channel = 0
<4>[ 5354.152557]       max scb = 2
<4>[ 5354.983978] wds: 0021 @ 01
<6>[ 5354.984100] qmi: wds: network stopped
<4>[ 5354.984375] wds: 0022 @ ff
<6>[ 5354.984588] qmi: wds: DISCONNECTED
<7>[ 5356.081939] ALS value: 0xB0, level: 5 #
<7>[ 5356.082519] set brightness = 164
<4>[ 5356.337585] Set auto channel = 1
<4>[ 5356.338806] wl_iw_setap: do passhash...
<4>[ 5356.437713] [00]: d8ce9015
<4>[ 5356.437835] [01]: 83fb21f6
<4>[ 5356.438049] [02]: c2a67115
<4>[ 5356.438171] [03]: bd9507c0
<4>[ 5356.438262] [04]: f9f16a42
<4>[ 5356.438385] [05]: a4308725
<4>[ 5356.438598] [06]: 92d86b98
<4>[ 5356.438720] [07]: fa184156
<4>[ 5356.438842] wl_iw_setap: passphase = 1590ced8f621fb831571a6c2c00795bd426af1f9258730a4986bd892564118fa
<4>[ 5356.450347] ap setup done
<4>[ 5356.452423] send AP_UP
<4>[ 5356.453399] penguin, get AP_ASSOC_LIST_GET
<6>[ 5356.454193] rmnet_stop()
<4>[ 5356.471343] wds: 0021 @ 01

Standard Initialization:
<4>[ 5542.693817] Dongle Host Driver, version 4.218.161.0
<3>[ 5545.224090] mmc0: Command timeout (cmd= 8 mci_st=00000004)
<4>[ 5545.232208] msmfb_start_dma 20.111 ms after vsync request
<7>[ 5545.240600] mmc0: queuing CIS tuple 0x91 length 3
<6>[ 5545.240783] mmc0: new high speed SDIO card at address 0001
<4>[ 5545.247802] DHD: dongle ram size is set to 294912(orig 294912)
<4>[ 5545.422363] wifi: Disable 802.11n
<4>[ 5545.519622] Enter set packet filter
<4>[ 5545.522216] start to add pkt filter 100
<4>[ 5545.530151] Enter set packet filter
<4>[ 5545.533538] start to add pkt filter 102

<4>[ 5545.540222] dhd_preinit_ioctls:str=0x00112233445538e7d819d70c0800450000000000000000000000000000000000000000000000000000000000

<6>[ 5545.542358] eth0 (): not using net_device_ops yet
<4>[ 5545.546325] eth0: Broadcom Dongle Host Driver mac=38:e7:d8:19:d7:0c

<4>[ 5546.694702] Enter set packet filter
<4>[ 5546.705078] Enter set packet filter
<4>[ 5546.707550] start to add pkt filter 104
<7>[ 5556.769165] eth0: no IPv6 routers present

From here we can workout that the best way to get access point functionality out of this guy is to use the ioctl found in the bravo source code. The system performs a very poorly documented system call to SIOCSIWPRIV, an Android specific enhancement to the standard wireless drivers in Linux, and specifically calls a  AP_PROFILE_SET from the pointer blob. This executes wl_iw_setap and from there the whole show starts.

The goal from here is to write a small C application that executes an ioctl to call this handle and initialize the AP hotspot mode, just as the sprint app does. Experimenting with iwconfig shows the _ap.bin firmware supports most of the functionality needed, but I think that in order to really knock it into AP mode the undocumented Android system call must be made. 

1
2
3
4
5
6
7
8

./iwconfig eth0
eth0      IEEE 802.11-DS  ESSID:"AndroidTether"  Nickname:""
          Mode:Master  Frequency:2.412 GHz  Access Point: 38:E7:D8:19:D7:0C
          Bit Rate=54 Mb/s   Tx-Power:32 dBm
          Retry min limit:7   RTS thr:off   Fragment thr:off
          Encryption key:off

#

Look promising? I'll keep you updated!

Permalink | Leave a comment  »

The EVO does not use the standard ADB drivers that come with the development kit. This can be a challenge for those of us who do not know where to look. Luckily HTC has included drivers for our devices in their HTC Sync program, installed on the SD card.

The easiest way to get up and running with ADB is to install the free HTC Sync utility that came with your phone. This opens up all kinds of fun possibility, especially when combined with an easy root technique. It will give you shell access to your phone and the ability to debug and test applications.

Getting up and Running

First you must install the HTC Sync utility from your included SD card. To do this plug your phone into your computer using the included USB cable, next pull down the notification bar and put it into Removable Storage mode. This will cause your phone to mount as a device in Windows.

Navigate to the HTC Sync directory and launch the included installer. Follow the steps and complete the wizard. Plug your phone into the computer at this point and it should automatically install all required drivers.

Enabling debugging on your phonee

Some of you may need to enable debugging over USB on your phone for ADB to properly launch. To do this on the device, go to the home screen, press MENU, select Applications > Development, then enable USB debugging.

You now have an ADB interface running!

Assuming you installed the Android Debug Bridge, outlined in my previous post, you can now call ADB from the command line. Simply press Windows Key + Run, type cmd, and press enter.

In this new window type adb shell and hit enter. Assuming your device is connected and operational you will be viewing a shell prompt on your phone. Where you go from here, is up to you!

As a final note I'll mention that if you're like me the default width of the command prompt will drive you absolutely crazy. To rectify this I recommend setting the width as shown:

Permalink | Leave a comment  »

I'll be honest, like many I am primarily a .Net developer. My favorite languages are VB and C# and I breath CIL. The world of Android is a little foreign to me and getting started seems like a challenge, however the more Android develops, the more I start to see similarities with .Net. Both languages have an extensive library of predefined functions to speed up development, both are what I call OO 2.0 languages, embodying and fully embracing object-oriented programming, and, with Froyo, both use just-in-time compilers to run the code.

Fortunately, unlike .Net, one of the requirements to get up and running with a full Android IDE is not a pocket full of bills (or enrollment in a major university's CS program). Android is developed mainly in Eclipse, a development environment that you will form a strong love/hate bond with as your relationship with it grows.

Installing Eclipse

Android is a java-based language and uses Eclipse. To start out we must install Eclipse. So here are the steps required to install:

1. Go to eclipse.org's downloads section.
2. Download the Java EE Edition. It is a zip file and you will need to extract it to your disk. I chose to extract it to C:\android for convenience.
3. Launch Eclipse, in this case from C:\android\eclipse\eclipse.exe (no install needed, +1), and set your default workspace.

Installing Android Development Kit

1. Go to "http://developer.android.com/sdk/index.html to find the latest version of the development kit.
2. Download the windows zip file and extract to your C:\android directory.
3. Add the newly extracted directory to your PATH
   • Right click on My Computer and go to Properties
   • Click on "Advanced System settings"
   • Open the Environment Variables tab. Scroll down in the System variables group until you find a variable called PATH. Click Edit..
   • In the Variable value text box append to the end ";C:\android\android-sdk-windows\tools" to add the Android tools directory to your PATH. This will make your life easier down the road when you just want to launch adb.exe without navigating to this directory.

Installing the Eclipse Android Plugin

1. Launch Eclipse as before.
2. Go to the Install New Software… menu
3. Click Add… and type in Google's Android update location https://dl-ssl.google.com/android/eclipse/
4. Grab the checkbox for Developer Tools". Click Next and Finish on the next screens until an install-like process begins.
5. Once the install finishes Eclipse will ask you to restart. Allow this to happen. Restarting is good.
6. When Eclipse restarted we must set the Android SDK location. To do this navigate to the preferences dialog.
7. Select Android and choose Browse… select your extracted directory C:\android\android-sdk-windows.
8. Choose your preferred development platform. Since I will be developing with the EVO I chose Android 2.1-update1.
9. Click ok. You're done!

At this point you have a fully functional build environment. From here you still have to prepare your emulator, run a sample project, and get started in the Android marketplace. As I work with my Android phone.

Permalink | Leave a comment  »

Today the mobile broadband industry took a step backwards as AT&T proved itself to be even more irrelevant by going back to capped data plans. From what I've gathered they plan to remove their $30 unlimited plan and more to a 2-tier approach to data:

Permalink | Leave a comment  »

Extracting useful data from T-SQL on the fly can be challenging. Writing GROUP BY statements can bend your mind and make life difficult. Like any good SQL programmer I've built up a tool chest of SQL queries over the years that serve me well in the field. I've had to sort, aggregate, count, average, and pivot data a countless number of times and reaching for the T-SQL documentation becomes a tedious task. One of my favorite queries, and one of the ones I see beginner SQL programmers struggle with most often, certainly has to be grouping a series of discrete, time-stamped events by days and pulling useful information from it. To this end I present my implementation of the T-SQL group-by day and month query. 

GroupEventsByDate.sql

1
2
3
4
5
6
7
8
9

SELECT 
	DatePart(month,[Date]) as lMonth, 
	DatePart(day,[Date]) as lDay,
	Count(*) as lCount
    FROM [MY].[dbo].[UsageLog] 
    Where [Date] > '12/31/2009'
    Group By DatePart(month,[Date]), DatePart(day,[Date]) 
    Order By lMonth Desc, lDay Desc
GO

Now, for some explanation

I've seen many SQLers try to create a string concatenated field from the day and month as a new column. This works but has both poor performance and making sorting nearly impossible. The solution is to GROUP BY multiple columns and select each one as a separate column. This allows us to do a traditional sort with days and months appearing in the right order. The current implementation is shorthanded, I use it to look at data over the past 12 months most often so I do not select the year as a unique identifier, however it should be clear how to accomplish this.

The final question you might have is how to include days which there was no official data for. T-SQL doesn't offer us a native way to do this so we must resort to a sort of hack. Here's the procedure you'd follow:

1. Create a time-dimension table and insert all the dates from the starting date to the current (and beyond). This is best accomplished via a script in your favorite language or this particularity nasty SQL hack, documented very well here.
2. RIGHT JOIN against the newly created time-dimension table with a GROUP BY clause to ensure all those 0 values are properly displayed

This method works great when the missing dates are mission critical and can't be added by a scripting language post-humorously.

Permalink | Leave a comment  »

Getting this guy to work with Windows 7 x64 is a pain, however it is one of the most economical programmers out there so I feel it was definitely worth the effort. It is based off of the libusb-32 driver package and, like many open-source projects with no cash flow, it has no signed driver support. To fix this involves some pretty nasty hacks, but luckily Microsoft has simplified the process for us with their bcdedit utility for editing Windows boot options. Follow the steps after the break and you'll be off programming AVRs before you know it!

Warning: Making changes to your computer can cause permanent, hard-to-fix problems. Proceed at your own risk!

1. Click on your Start menu and type "cmd".
2. Right click on command prompt and goto "Run as Administrator".
3. Execute the following commands: `bcdedit -set loadoptions DISABLE_INTEGRITY_CHECKS` and `bcdedit -set TESTSIGNING ON`
4. Reboot and install the drivers from SparkFun's site.
5. Program away!

As a side effect your computer desktop will now permanently display "Test Mode"

To undo these procedures simply run the commands again, substituting `-set` for `-deletevalue`.

Permalink | Leave a comment  »

In .Net the defining feature of the libraries is that there is at least 2 different ways you can perform any one task. Each one usually has it’s own subtle uniqueness to it but in the end they all get you from Point A to Point B. The problem is figuring out which approach works best and combining all the different framework elements into a coherent application. In my line of work I write a lot of middlewear. Applications that interfaces with systems made before XML was a blip on the horizon and basically act as glue. One of the most common ways I interact with these applications is by reading their log files and looking for key phrases and elements. I have written more log-watchers than I can count so I’ve finally made the end-all piece of code to watch a log file.

In writing this piece of code you might instantly light up and remember that .Net supports event-driven file watching. There is a very nice IO.FileSystemWatcher class out there that does in fact provide most of the functionality required to implement this design. I used this class for about a week myself. What I came to find is that it is simply riddled with bugs, especially when watching files over the network. The object will simply loose touch with the network folder you are watching. This resulted in a mysterious mandelbug that appears only when mission-critical files need processing. The event-firer will grow eerily quite and nothing will ever be watched again. This is a problem. Upon ripping the FileSystemWatcher from the code and replacing it with a simple thread-based polling piece of code the problem was solved. The application that runs this code is now one of the most stable in the building, with an uptime approaching 2 years and counting.

So without further discussion here is my code for a simple watcher class that will watch either a directory with rolling log files or a single file.

WatcherPhrase.vb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

Public Class WatcherPhrase
    Private _name As String
    Private _regex As Regex

    Public ReadOnly Property Name As String
        Get
            Return _name
        End Get
    End Property

    Public ReadOnly Property Matcher As Regex
        Get
            Return _regex
        End Get
    End Property

    Public Sub New(ByVal Matcher As Regex, ByVal Name As String)
        _name = Name
        _regex = Matcher
    End Sub
End Class

FileWatcherEventArgs.vb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

Public Class FileWatcherEventArgs
    Inherits EventArgs

    Private _matcher As WatcherPhrase
    Private _file As IO.FileInfo
    Private _match As String

    Public ReadOnly Property matcher As WatcherPhrase
        Get
            Return _matcher
        End Get
    End Property
    Public ReadOnly Property file As IO.FileInfo
        Get
            Return _file
        End Get
    End Property
    Public ReadOnly Property match As String
        Get
            Return _match
        End Get
    End Property

    Public Sub New(ByVal matcher As WatcherPhrase, ByVal file As System.IO.FileInfo, ByVal match As String)
        _matcher = matcher
        _file = file
        _match = match
    End Sub
End Class

FileWatcher.vb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118

Public Class FileWatcher
    Public Enum inputTypes
        File
        Folder
    End Enum

    Private _inputPath As String
    Private _inputType As inputTypes

    Private _isRunning As Boolean = False
    Private _stoping As Boolean = False
    Private _workerThread As System.Threading.Thread
    Private _matcherList As New List(Of WatcherPhrase)

    Public Event matchFound(ByVal sender As Object, ByVal e As FileWatcherEventArgs)

    Public Sub New(ByVal inputPath As String, ByVal inputType As inputTypes)
        _inputPath = inputPath
        _inputType = inputType
    End Sub

    Public Sub New(ByVal inputPath As String, ByVal inputType As inputTypes, ByVal matcher As WatcherPhrase)
        _inputPath = inputPath
        _inputType = inputType
        _matcherList.Add(matcher)
    End Sub

    Public ReadOnly Property isRunning As Boolean
        Get
            Return _isRunning
        End Get
    End Property

    Public Property matcherList As List(Of WatcherPhrase)
        Get
            Return _matcherList
        End Get
        Set(ByVal value As List(Of WatcherPhrase))
            _matcherList = value
        End Set
    End Property

    Public Sub Start()
        _workerThread = New System.Threading.Thread(AddressOf Process)
        _workerThread.Start()
        _isRunning = True
    End Sub

    Public Sub StopMe()
        _stoping = True
        If _workerThread IsNot Nothing AndAlso _workerThread.IsAlive = True Then
            _workerThread.Join()
        End If
        _isRunning = False
    End Sub

    Private Sub Process()
        Dim lastFile As IO.FileInfo
        Dim lastPosition As Long
        Dim currentFile As IO.FileInfo

        If _inputType = inputTypes.File Then
            currentFile = New IO.FileInfo(_inputPath)
        End If

        While True
            If _stoping = True Then
                Return
            End If

            ' Main loop
            If _inputType = inputTypes.Folder Then
                Dim dirSearch As New IO.DirectoryInfo(_inputPath)
                Dim mostModifiedFile As IO.FileInfo = (From c In dirSearch.GetFiles Order By c.LastWriteTime Descending).FirstOrDefault
                If lastFile Is Nothing Then
                    lastFile = mostModifiedFile
                    lastPosition = mostModifiedFile.Length - 1
                End If

                If currentFile IsNot Nothing Then
                    ' If the last modified file has changed we assume a new logfile has been created. 
                    If lastFile.FullName <> mostModifiedFile.FullName Then
                        lastFile = mostModifiedFile
                        lastPosition = 0
                    End If
                End If

                currentFile = mostModifiedFile
            End If

            Try
                If currentFile IsNot Nothing AndAlso lastPosition < currentFile.Length Then
                    Dim oFile As New IO.FileStream(currentFile.FullName, IO.FileMode.Open, IO.FileAccess.Read, IO.FileShare.ReadWrite)
                    oFile.Seek(lastPosition, IO.SeekOrigin.Begin)
                    Dim oReadStream As New IO.StreamReader(oFile)
                    Dim str As String = oReadStream.ReadToEnd

                    If str.Contains(vbCrLf) = True Then
                        Dim list As New List(Of String)(str.Substring(0, str.LastIndexOf(vbCrLf)).Split(New Char() {vbCrLf}))
                        For Each oEl As String In list
                            For Each oReg As WatcherPhrase In _matcherList
                                Dim matchResult = oReg.Matcher.Match(oEl)
                                If matchResult.Success = True Then
                                    RaiseEvent matchFound(Me, New FileWatcherEventArgs(oReg, currentFile, oEl))
                                End If
                            Next
                        Next
                        lastPosition = lastPosition + str.LastIndexOf(vbCrLf) + 1
                        lastFile = currentFile
                    End If
                End If
            Catch ex As Exception

            End Try
        End While
    End Sub
End Class

And how does it all fit together? Here is a very simple example of the classes' usage.

FileWatcherTest.vb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

Public Class Tester
    Private WithEvents _fileWatcher As FileWatcher
    Public Sub StartTest()
        Dim oMatchPhrase As New WatcherPhrase(New System.Text.RegularExpressions.Regex("An error has occured."), "Error Log Pattern Matcher")
        _fileWatcher = New FileWatcher("C:\logs", FileWatcher.inputTypes.Folder, oMatchPhrase)
        _fileWatcher.Start()
    End Sub

    Private Sub HandleFileWatcher(ByVal sender As Object, ByVal e As FileWatcherEventArgs) Handles _fileWatcher.matchFound
        Debug.WriteLine("Match Found: " & e.matcher.Name & " found a match in " & e.file.Name & " at """ & e.match & """")
    End Sub

    Public Sub EndTest()
        _fileWatcher.StopMe()
    End Sub
End Class

And there you go, one fully functional log watcher with each and every bug worked out. The following code is in use every day with hundreds of log files over many different testing expressions!

Permalink | Leave a comment  »

In my current project I am implementing a simple data logger. When the hardware was designed we did not implement a nice RTC module, such as the chip used in Sparkfun’s clock module here: http://www.sparkfun.com/commerce/product_info.php?products_id=99. In lieu of such a nice solution I have opted to implement a basic, semi-accurate clock using an 8-bit timer in the AVR. Here’s the steps:

1. Calculate from the clockspeed the interval and width that timer needs to fire at to record a 1Hz measurement
2. Enable timer interrupts
3. Capture the timer interrupt and increment a 4-byte integer

In my case the microcontroller is running off of a 8Mhz crystal. Setting the timer prescaler to clk/256 produces 31,250 clock pulses. Setting OCRA to 250 and counting up to 124 before incrementing the second produces 31,250/250/124 = 1 pulse per second.

For this project I have decided to store the time as the number of seconds elapsed since January 1st, 2001. This is convenient because the host system can handle all the nasty stuff such as leap-years and other oddities in the calendar while the microcontroller just stores a number.

And without further ado I shall present some code!

AVR timer.c

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

volatile char counter = 0;
volatile long seconds = 0;

ISR(TIMER0_COMPA_vect)
{
	counter++;
	if (counter==124)
	{
	seconds+=1;
	counter=0;
	}
}

void timer_init()
{
	TIMSK0 = 0b00000010;  // Enaled Timer CompA interupt
	TCCR0B = 0b00000100;  // Set speed to clk/256
	TCCR0A = 0b00000010;  // Set timer to CTC mode
	OCR0A = 250;		  // Set timer period
}

In addition to this AVR code some host-side code is needed to set and read the time from the AVR accurately. For this I have used VB.Net.

Generating appropriate times

1
2
3
4
5
6
7
8
9
10

Public Sub TestRoutines()
' Setting the time
      Dim StartTime As DateTime = New DateTime(2000, 1, 1, 1, 0, 0)
      Dim SecTimeSpan As TimeSpan
      SecTimeSpan = Now() - StartTime
      Dim _bytes() As Byte = BitConverter.GetBytes(CType(SecTimeSpan.TotalSeconds, UInt32))

' Converting received time into Date obj
     Dim deviceDate as New Date(2000, 1, 1, 1, 0, 0).AddSeconds(BitConverter.ToUInt32(_bytes, 0))
End Sub

Testing has shown this clock to drag a millisecond or two for every minute of operation. Not the greatest performance but for a data logger it is adequate.

Permalink | Leave a comment  »

While .Net is great for developing large, enterprise-level applications I have found it to be lacking basic manipulation abilities. When developing applications that involve low-level microcontrollers I often have struggled to find the correct syntax to do proper bit manipulation. Furthermore debugging byte arrays can get plain frustrating. As you are probably aware, the IDE displays bytes as numeric values (0-255) instead of the standard hexadecimal notation. This is fine and dandy but the majority (>99%) of technical documentation has chosen hexadecimal notation for representation of byte streams in protocols and interfaces. Translating can get tedious for those who do not speak fluent hex, a group of individuals that I currently belong to.

The .Net library is great but I feel it is lacking some easy-to-use tools for bit-manipulation. Using Visual Basic of course doesn't help things. To solve these problems I have created a set of basic bit-manipulating and debugging tools that I include in my base classes in order to output hexadecimal strings of byte arrays and to flip bits as needed. I have found these utilities invaluable when building low-level interface code that talks via USB or serial to devices that don't support the latest XML object serialization techniques for remoting cutting-edge .Net entity framework model do-dads and devices that don't have the luxury to run the full CLI.

Polling the Status of a Bit

1
2
3

    Protected Function GetBit(ByVal Byte As Byte, ByVal BitPosition As Integer) As Boolean
        Return ((Byte And 2 ^ BitPosition) <> False)
    End Function

Setting a bit

1
2
3
4
5
6
7
8
9

    Protected Sub SetBit(ByRef oByte As Byte, ByVal BitPosition As Integer, ByVal NewPosition As Boolean)
        Dim newByte As Byte = oByte
        If ((oByte And 2 ^ BitPosition) <> False And NewPosition = False) Then
            newByte = oByte Xor 2 ^ (BitPosition)
        ElseIf ((oByte And 2 ^ BitPosition) = False And NewPosition = True) Then
            newByte = oByte Or 2 ^ (BitPosition)
        End If
        oByte = newByte
    End Sub

Converting to ASCII Hexadecimal Representation from Bytes

1
2
3
4
5
6
7
8
9
10
11

    Public Function ByteToHex(ByVal comByte As Byte()) As String
        'create a new StringBuilder object
        Dim builder As New System.Text.StringBuilder(comByte.Length * 3)
        'loop through each byte in the array
        For Each data As Byte In comByte
            builder.Append(Convert.ToString(data, 16).PadLeft(2, "0"c).PadRight(3, " "c))
            'convert the byte to a string and add to the stringbuilder
        Next
        'return the converted value
        Return builder.ToString().ToUpper()
    End Function

Converting from ASCII Hexadecimal Representation to Bytes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

    Public Function HexToByte(ByVal msg As String) As Byte()
        If msg.Length Mod 2 = 0 Then
            'remove any spaces from the string
            Dim _msg As String
            _msg = msg
            _msg = msg.Replace(" ", "")
            'create a byte array the length of the
            'divided by 2 (Hex is 2 characters in length)
            'Dim comBuffer As Byte() = New Byte(_msg.Length / 2 - 1) {}
            Dim comBuffer((_msg.Length / 2) - 1) As Byte
            For i As Integer = 0 To _msg.Length - 1 Step 2
                comBuffer(i / 2) = CByte(Convert.ToByte(_msg.Substring(i, 2), 16))
            Next
            'loop through the length of the provided string
            'convert each set of 2 characters to a byte
            'and add to the array
            'return the array
            Return comBuffer
        Else
            Dim ABuffer() As Byte = {0, 0}
            Return ABuffer
        End If
    End Function

Hopefully you will find these functions as useful as I have over the years!

Permalink | Leave a comment  »

As part of my new commitment to blogging and sharing my knowledge with the outside world one of my first goals is to tune the tools needed to create professional, easy-to-read blog articles. Like many bloggers I do not want to have to worry about the format or the administration behind the blog, I want to create content.

To that end I have successfully figured out how to highlight syntax on Posterous:

1. Take your code and import it into your favorite text editor ( Editpad Pro is my favorite)
2. Replace all line breaks with <br />. To do this in Editpad simply use their powerful Find and Replace dialog box to select all line breaks and replace them with a HTML equivalent.
3. Surround your code in [ code] tags. You can specify a list of supported languages to achieve the formatting you desire. In SyntaxHighligher, the JavaScript-based engine powering Posterous's highlighting these language formats are called Brushes. A good list is available here: http://alexgorbatchev.com/wiki/SyntaxHighlighter:Brushes
4. Paste this muddy collection of code, HTML, and text into the Posterous post editor, in plain-text mode. 
5. You're done!

This process will produce professional-looking syntax highlighting and create greater visibility for your readers. 

Permalink | Leave a comment  »

Trudging through object-oriented forest leads to some interesting results. The beauty of a strongly-typed language must also be it's biggest downfall. In VB.NET in order to implement a polymorphic design pattern on a sensor class I had to learn the magic of DirectCasting an object. See the example below:

  Public Class Form1     Dim WithEvents oBaseClass As BaseClass     Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load         oBaseClass = New ExtendedClass         oBaseClass.Test() 'This doesn't work because the object was type casted.         End Sub       Private Sub TestEventHdlr() Handles oBaseClass.TestEvent         MsgBox("Event Fired")     End Sub End Class   Public Class BaseClass     Public Event TestEvent() End Class   Public Class ExtendedClass     Inherits BaseClass     Public Sub Test()         MsgBox("Test")     End Sub End Class

Surprisingly this does not work!

The issue lies in direct-casting. In order to use extended functions inside an object one must direct cast it, like this:

Public Class Form1 Dim WithEvents oBaseClass As BaseClass 'Early bound'

Permalink | Leave a comment  »