The ABA file format, sometimes called Direct Entry or Cemtext, is an unpublished de facto standard for bulk electronic payments. It is used by all major Australian financial institutions to interface with business accounting systems, and is supported by many popular accounting packages such as MYOB. For small-to-medium businesses, the most common usage for an ABA file is to upload it via their Internet banking portal – to avoid manually entering transaction details into the web application. As the name suggests, the format is dictated by the Australian Banking Association, and also has ties with the Australian Payments Clearing Association (APCA).

I was recently tasked with writing an exporter from a proprietary, relational database system to the ABA format. Since the specifications for the file format are not in the public domain (at least, not from an official source), I had to rely on some third-party sources to arrive at a solution. In this article, i’ll share those resources with you, as well as providing my own explanation of some of the finer points of the standard – to help you avoid common pitfalls when working with the format.

Overview of the ABA format

The ABA format is somewhat archaic; certainly based on outdated technology. It’s a plain-text format that uses fixed-width fields (most of which are severely constrained, to the point that most data going into them must be truncated or abbreviated in order to fit). It is not supposed to be human-readable, although a trained eye can interpret it. Its values are untyped and there is nothing to validate the correctness of an ABA file if you are not privy to the rules.

An ABA file consists of:

  • One ‘type 0’ record (i.e. line) which describes and provides instructions processing instructions for the batch
  • One or more ‘type 1’ records which provide the detail for each payment in the batch
  • One ‘type 7’ record which confirms the size and totals for the batch

(i.e. an ABA file is at least 3 lines long.)

Within each record/line in the file, and depending on the record type, there are a number of fixed-width fields which must be populated. Some are padded with zeroes, others with spaces. Some occupy the left portion of the field, others (like numeric fields) are right-justified. A complete line must consist of exactly 120 characters (excluding the CR/LF terminator). It’s helpful to write a function to perform the necessary padding/truncation and justification:

// requires a reference to System.Linq
string ToFixedWidth(string input, int length, char padding = ' ', bool rightJustify = false) {
	char[] output = new char[length];
	if (rightJustify) input = new string(input.Reverse().ToArray());

	for (int i = 0; i < length; i++) {
		if (i < input.Length)
			output[i] = input[i];
		else
			output[i] = padding;
	}

	if (rightJustify) output = output.Reverse().ToArray();
	return new string(output);
}

// to pad a string out to 18 characters, left-justified and padded with spaces:
ToFixedWidth(str, 18);

// to pad a decimal number out to 10 characters, right-justified and padded with zeroes:
ToFixedWidth((num * 100).ToString("0000000000"), 10, '0', true);

Some additional notes:

  • Most examples i’ve seen capitalise all text fields, however this is not necessary. Some financial institutions will retain the casing of your narrations/annotations, if only on the sender’s bank statement.
  • All dollar amounts are expressed in whole cents, with no digit grouping. Multiply your decimal figures by 100 and use an appropriate number format string (e.g. “0000000000”).

Resources for the file format specification

The following are two excellent resources for the specification of the ABA file format itself:

Between these two documents, you should have all the information you need to construct the ABA records themselves. There is some inherent ambiguity in these, however, which is why i’d like to add the following notes:

Descriptive record

Reel sequence number

There is a maximum number of payments you can include per ABA file, if only because the dollar amount and number of records fields are of a fixed length. If you need to split your batch into multiple files, the reel sequence number identifies each one. However, in the vast majority of cases, you will only have one file, numbered 01. You do not have to keep track of which numbers you have used previously.

User Preferred Specification

This is one of two pieces of information that identify the user and their financial institution. Some banks will dictate the value for this field, others are less restrictive and will permit any form of the name on the user’s bank account. It’s best to make the value for this field user-maintainable.

User Identification Number

Contrary to its description, this number identifies the financial institution, not the user. Each FI has its own number (or possibly several, depending on the size of the bank) which is issued by APCA. This number does not change between batches; it’s tied to the bank and, potentially, the account. Most banks document this number somewhere in their internet banking portal instructions.

Detail records

Indicator

As far as I can tell, this field can be used to indicate that a particular record represents a change to a prior record in the batch, or to existing details held on file for the payee. It’s also clear that not every financial institution pays attention to this field. Unless you are dealing with thousands of payments at a time, you should leave this field blank to make it clear that each record represents a separate payment.

Transaction code

In most cases, you should use the code 50, which represents a non-specific credit to the bank account. For payroll transactions, use the code 53. The other transaction codes appear to be of relevance to the ATO and superannuation funds only.

Lodgement reference

This is the text that appears on the payee’s bank statement.

Trace record (BSB and Account No)

This is the BSB and Account Number of your bank account.

Name of remitter

If your bank supports it, this lets you track the name of the person in your organisation who authorised the payment.

Final words

Hopefully, this points you in the direction of the right resources and helps to dispel some of the confusion surrounding the ABA file format. By following the specification exactly, you will produce a file that can be read by all major Australian banks and has the potential to save time and reduce errors when processing large numbers of payments.

See also

ABA/Cemtext Validator – An ABA file validator and SDK based on this article

Windows Forms provides many mechanisms for building professional custom UI controls that match the operating system style; by combining visual style renderers, system colours/brushes, the ControlPaint class and more, it’s possible to reproduce most of the standard Windows controls in user code.

There is, however, one aspect of the built-in controls that can be difficult to recreate in managed code: Starting with Windows Vista, fade animations are used for many controls (e.g. Button, ComboBox, TextBox, etc) when transitioning between states such as focus, mouse-over and button pressing. Internally, these animations are handled by the buffered paint API (part of uxtheme.dll, the library responsible for visual styles).

Most developers would be content with instantaneous visual state changes, but to the trained eye, the lack of smooth transitions can really make a custom control stand out from a built-in one. The good news is that, although there is no managed API for buffered painting, it’s relative easy to harness using PInvoke.

Buffered Paint API – The Basics

Imports

[DllImport("uxtheme")]
static extern IntPtr BufferedPaintInit();

[DllImport("uxtheme")]
static extern IntPtr BufferedPaintUnInit();

[DllImport("uxtheme")]
static extern IntPtr BeginBufferedAnimation(
    IntPtr hwnd,
    IntPtr hdcTarget,
    ref Rectangle rcTarget,
    BP_BUFFERFORMAT dwFormat,
    IntPtr pPaintParams,
    ref BP_ANIMATIONPARAMS pAnimationParams,
    out IntPtr phdcFrom,
    out IntPtr phdcTo
);

[DllImport("uxtheme")]
static extern IntPtr EndBufferedAnimation(IntPtr hbpAnimation, bool fUpdateTarget);

[DllImport("uxtheme")]
[return: MarshalAs(UnmanagedType.Bool)]
static extern bool BufferedPaintRenderAnimation(IntPtr hwnd, IntPtr hdcTarget);

[DllImport("uxtheme")]
static extern IntPtr BufferedPaintStopAllAnimations(IntPtr hwnd);

Usage

  • You initialise/end a session (typically lasting for the life of the control) using BufferedPaintInit/BufferedPaintUnInit.
  • You begin a fade animation with BeginBufferedAnimation, passing in a BP_ANIMATIONPARAMS structure to describe the transition. A handle to the animation and two empty bitmaps are returned.
  • You paint the start and end frames onto the respective bitmaps (using GDI+, as if you were painting the control normally) and call EndBufferedAnimation to start playing it.
  • While the animation is playing, the control’s Paint event will be fired multiple times. Check to see whether it was triggered by the buffered paint animation by checking the value returned by BufferedPaintRenderAnimation; if so, don’t paint the control yourself.
void Control_Paint(object sender, PaintEventArgs e) {
    IntPtr hdc = e.Graphics.GetHdc();
    if (hdc != IntPtr.Zero) {
        // see if this paint was generated by a soft-fade animation
        if (!Interop.BufferedPaintRenderAnimation(Control.Handle, hdc)) {
            BP_ANIMATIONPARAMS animParams = new BP_ANIMATIONPARAMS();
            animParams.cbSize = Marshal.SizeOf(animParams);
            animParams.style = BP_ANIMATIONSTYLE.BPAS_LINEAR;

            // set duration according to state transition
            animParams.dwDuration = 125;

            // begin the animation
            Rectangle rc = Control.ClientRectangle;
            IntPtr hdcFrom, hdcTo;
            IntPtr hbpAnimation = Interop.BeginBufferedAnimation(
                Control.Handle,
                hdc,
                ref rc,
                BP_BUFFERFORMAT.BPBF_COMPATIBLEBITMAP,
                IntPtr.Zero,
                ref animParams,
                out hdcFrom,
                out hdcTo
            );

            if (hbpAnimation != IntPtr.Zero) {
                if (hdcFrom != IntPtr.Zero) /* paint start frame to hdcFrom */;
                if (hdcTo != IntPtr.Zero) /* paint end frame to hdcTo */;
                Interop.EndBufferedAnimation(hbpAnimation, true);
            }
            else {
                /* paint control normally */
            }
        }

        e.Graphics.ReleaseHdc(hdc);
    }
}

Some further notes about using the buffered paint API on Windows Forms controls:

  • Animations are not supported if the control is double-buffered (DoubleBuffered property set to true or OptimizedDoubleBuffer style flag set).
  • To reduce flickering, override the control’s OnPaintBackground method and don’t call the base class method. You can paint the control’s background manually when you paint the rest of it.
  • Whenever the control is resized, all running animations should be stopped using BufferedPaintStopAllAnimations.

BufferedPainter – A Managed Class to Simplify Buffered Painting

Buffered painting involves a certain amount of boilerplate code. You need to add code to the control’s creation/disposal events, override the Paint event using a particular pattern and then provide an alternative method for painting the control (either to the screen or to a bitmap).

One way to eliminate this boilerplate code would be to write a base class, derived from Control, which provided this functionality. This is somewhat limiting, however, as all custom controls using buffered painting would have to inherit from this class. In reality, you are likely to want to use buffered painting both in completely new controls and when subclassing existing controls (e.g. ComboBox). For this reason, i’ve written a class which sits in isolation and attaches to any type of control.

BufferedPainter is a generic class which allows any type to be used to represent a control’s visual state; this may be an enumeration, an integer or even a more complex type. As long as the type provides an Equals method (or has a suitable default implementation), it can be used to track state transitions. A simple button control might have three states; Normal, Hot and Pushed. BufferedPainter holds information about state changes and the duration of animations between states (where desired). It stores the current visual state of the control, overrides the control’s Paint event and provides a PaintVisualState event to be handled by user code.

It also provides a mechanism to simplify the process of triggering changes in the control’s visual state; in addition to manually setting the state of the control (using the State property), you can add a trigger which changes to a particular state in response to a condition (such as the mouse being over the control). This further reduces the amount of code necessary in the control. Conditions can be specific to a region within the control’s bounds, and anchoring can be used to automatically update the region when the control is resized.

Adding buffered paint support to a control is as simple as:

// using an enum type 'MyVisualStates' to describe the control's visual state
BufferedPainter<MyVisualStates> painter = new BufferedPainter<MyVisualStates>(/* control instance */);
painter.PaintVisualState += /* event handler which paints the control in a particular state */;

// describe the state transitions we want to animate
painter.AddTransition(MyVisualStates.Normal, MyVisualStates.Hot, 125); // fade in
painter.AddTransition(MyVisualStates.Hot, MyVisualStates.Pushed, 75);
painter.AddTransition(MyVisualStates.Hot, MyVisualStates.Normal, 250); // fade out

// describe what causes the control to change its visual state
painter.AddTrigger(VisualStateTriggerTypes.Hot, MyVisualStates.Hot); // mouse over
painter.AddTrigger(VisualStateTriggerTypes.Pushed, MyVisualStates.Pushed); // mouse down

Download

BufferedPainting.zip (includes example control)

Final Words

The buffered paint API closes one of the final gaps in writing custom controls in managed code which match the OS style, both in terms of static appearance and animation. I hope you find my code useful in implementing smooth transitions in your own custom controls.

In C#, when we think of member-wise equality, we often think of value types. Indeed, their default behaviour (in contrast to that of reference types) is to provide both Equals() and GetHashCode() methods based on their composition rather than their location in memory. However, there are times when we want our reference types to be regarded as equal based on composition as well; the most obvious example of this is System.String, which masquerades as a value type in more ways than one. Let’s look at some other examples of reference types that we might want to equate in a member-wise fashion:

  • Database objects; where multiple instances of the same record should be regarded as equal.
  • An in-memory tuple; whose values are of importance and their provenance not.
  • A compound key in a dictionary or table; where it must be possible to create a new, separate instance to match an existing one.

It’s also worthwhile to point out that anonymous types exhibit member-wise equality as well; this makes it possible to do such things in LINQ as performing a group-by on multiple fields.

Needless to say, there are times when we want this behaviour in our reference types. The usual method of implementing this behaviour (unless using an anonymous type, of course) is to override the Equals/GetHashCode methods in a similar manner to this:

public override bool Equals(object obj) {
    if (obj is MyType) {
        MyType that = (MyType)obj;
        return (this.PropertyA == that.PropertyA) && (this.PropertyB == that.PropertyB);
    }
    else {
        return base.Equals(obj);
    }
}

public override int GetHashCode() {
    return PropertyA.GetHashCode() ^ PropertyB.GetHashCode();
}

Note: It’s important to override both methods and be consistent in the logic used to determine equality/identity for the type. Some framework methods will call the full-blown Equals() method, while others will be content to operate on the hash code only. As a developer, you expect to get the same behaviour every time, without having to know how the framework methods are implemented!

As you can see, this is typical boilerplate code; there seems to be no obvious way to avoid having to type this code out every time, since each class will have a different set of properties (or other members) that must be considered for equality… or is there? What if we could write a very clever base class from which to extend any reference types we want to equate in a member-wise fashion? A base class that would somehow know the public properties of whichever subclass was being compared at the time?

I Know What You’re Thinking…

…you’re thinking reflection, aren’t you? Well, true, reflection does allow a base class to discover the properties of its children, and yes, it could pop the values out of each property of each instance and try to equate them… but… and this is the big BUT… this would be a terribly inefficient technique!

public override bool Equals(object obj) {
    // we're not even checking whether 'obj' is of the same type
    foreach (PropertyInfo property in this.GetType().GetProperties()) {
        if (!property.GetValue(this, null).Equals(property.GetValue(obj, null))) return false;
    }

    return true;
}

Even if we cached the complete set of PropertyInfo objects for each subclass (so they would only have to be retrieved once per type), there’s still the massive overheads of using the reflection API to dynamically get the property values – something which must be performed during every call to Equals() and GetHashCode(). (Think about it – there’s no other way to get the property values. We can’t just cast to the subclass type, because we don’t know what the type is! It might not even exist at the point when the base class is compiled.)

No, i’m afraid reflection doesn’t address this need adequately. In my tests, a reflection-based Equals method like the one above performed at roughly 16 times slower than a type written using boilerplate code!

Expression Trees to the Rescue!

The poor performance of the reflection technique comes largely from the inability to ‘teach’ the runtime how to efficiently repeat the equality comparison once it has discovered the properties of the class; it never learns and it never gets faster! However, there is a different technique we can use which allows the runtime to discover how to perform an equality comparison efficiently after learning the process once; we can build an expression tree to model the equality comparison:

Expression tree for member-wise Equals method

Expression trees are not only a means of representing lambda expressions in a dynamic, structured manner – their real power comes from the ability to dynamically construct an expression tree using the API, then compile it into bona fide, executable code – using Expression<T>.Compile(). What expression trees allow us to do in the context of our base class, then, is to dynamically create (and then compile) a tailor-made Equals() and GetHashCode() implementation for each subclass – and then call these methods again and again and again, with very little performance hit!

static Func<object, object, bool> MakeEqualsMethod(Type type) {
    ParameterExpression pThis = Expression.Parameter(typeof(object), "x");
    ParameterExpression pThat = Expression.Parameter(typeof(object), "y");

    // cast to the subclass type
    UnaryExpression pCastThis = Expression.Convert(pThis, type);
    UnaryExpression pCastThat = Expression.Convert(pThat, type);

    // compound AND expression using short-circuit evaluation
    Expression last = null;
    foreach (PropertyInfo property in type.GetProperties()) {
        BinaryExpression equals = Expression.Equal(
            Expression.Property(pCastThis, property),
            Expression.Property(pCastThat, property)
        );

        if (last == null)
            last = equals;
        else
            last = Expression.AndAlso(last, equals);
    }

    // call Object.Equals if second parameter doesn't match type
    last = Expression.Condition(
        Expression.TypeIs(pThat, type),
        last,
        Expression.Equal(pThis, pThat)
    );

    // compile method
    return Expression.Lambda<Func<object, object, bool>>(last, pThis, pThat)
        .Compile();
}

Of course, we must still use reflection when constructing the expression trees – how else will we discover the properties? But the key point is that we only have to do this step once; from then on, we have a compiled method (callable via a delegate) which is almost indistinguishable from one written using the boilerplate technique. The method takes parameters of type System.Object, because it would be impossible to cast to the subclass type (or write the delegate signature for it) if it required strongly-typed parameters.

The process for creating the GetHashCode() method is fairly similar; we simply replace the conditional AND nodes with XOR, and the Equals nodes with calls to each property’s own hash code method. Obviously, there’s only one input in this case.

We store the delegates for each subclass type against their corresponding System.Type in a class-scoped Dictionary<TKey,TValue>. When an instance is constructed, it checks to see if its type is in the dictionary; if not, it generates and compiles the methods, otherwise it simply proceeds. The delegates are invoked thusly:

public override int GetHashCode() {
    return _functions[GetType()].GetHashCodeFunc(this);
}

public override bool Equals(object obj) {
    return _functions[GetType()].EqualsFunc(this, obj);
}

Usage

With the base class written, all that is needed in order to access the member-wise equality functionality is to extend the class and add an arbitrary number of public properties (only a get accessor is required). For example:

public class MyMemberwiseType : MemberwiseEqualityObject {
    public Guid ID { get; set; }
    public int Number { get; set; }
    public DateTime Date { get; set; }
}

Any attempt to equate two instances of MyMemberwiseType will result in a member-wise comparison. Likewise, operations using the hash code (such as Enumerable.GroupBy) will also operate in this fashion.

Performance

I ran a simple test which compares the execution time of three different memberwise equality comparison techniques (anonymous types, reflection and my expression tree implementation). Each subclass had 8 properties, with each test instance being initialised with the same set of random values (making the test completely fair on each implementation). Under these conditions, when faced with the task of comparing 5 million instances for equality, I obtained the following results:

  • Using an anonymous type: 0.770 sec – this is the fastest, obviously, since its Equals() method is generated at compile-time.
  • Using reflection: 9.882 sec
  • Using dynamically-compiled equality comparison: 0.972 sec

As you can see, there is a massive overhead in using reflection, however my dynamic compilation technique produces results which are much more comparable to a hard-coded method. There’s no such thing as a free lunch, as they say, but in this case the sacrifice is only a small one to avoid a lot of boilerplate code!

Final Words

Currently, the base class performs its comparison on all public properties, unconditionally. It would not be difficult to use a custom attribute type to include/exclude members from consideration, or even to allow non-public properties or fields to be included. Since these changes would affect only the structure of the expression tree (and not add any processing to the Equals() method itself), there would be no performance hit in doing so.

In summary, then, i’ve demonstrated how to save on boilerplate code for member-wise equality comparisons by writing a base class which dynamically generates Equals() and GetHashCode() methods, using expression trees. If, like me, you find the task of writing such code to be a tedious exercise, I hope you enjoy the possibilities presented by this solution 🙂

Download

MemberwiseEquality.zip – includes MemberwiseEqualityObject (the implementation itself), ReflectEqualityObject (for performance comparison) and a console application to replicate my tests.

Okay, so you’ve got a database full of people and their birthdays, and you’d like to calculate someone’s age. Would you, like me, immediately look to the Transact-SQL DATEDIFF function as a means of quickly and painlessly performing this calculation? That’s what I thought!

This is the statement that you would expect to do the job for you:

SELECT DATEDIFF(yy, @BirthDate, GETDATE());

Now, imagine your surprise when, instead of getting the behaviour you expect (i.e. the difference between the two dates, expressed in years), you get a number that, until the person’s birthday falls for the current year, is always 1 too large?

If that’s the case, you’ve fallen into the same trap that I did. If you read into it, you’ll find that DATEDIFF doesn’t really calculate the difference between two dates at all. Instead, it counts the number of date and time boundaries crossed between the two dates specified.

This means that, the larger the increment you ask for (i.e. years in this case), the less precise the function is in the calculation of its result. In fact, the statement above is equivalent to:

SELECT YEAR(GETDATE()) - YEAR(@BirthDate);    -- produces the same result

…which we know is definitely not the correct way to calculate a person’s age!

We run into the same problem if we ask for the results expressed in terms of months instead of years (and then divide by 12 to get an age in years), but the problem only occurs for the first part of the month in which the person’s birthday falls. If we try to get the result in days instead of months or years, we have to start accounting for leap years and the solution becomes too complex.

The easiest way to calculate a person’s age in SQL, therefore, it to use some simple conditional logic based on the day, month and year components of the date of birth:

DECLARE @Age int;
SET @Age = YEAR(GETDATE()) - YEAR(@BirthDate);
IF (MONTH(GETDATE()) < MONTH(@BirthDate)) OR (MONTH(GETDATE()) = MONTH(@BirthDate) AND DAY(GETDATE()) < DAY(@BirthDate))
    SELECT @Age - 1;
ELSE
    SELECT @Age;

Complete Function

I often deal with information about people who may either be living or deceased. For deceased persons, we’re still interested in calculating age, but we want their age as at their date of death. It’s also useful to calculate a person’s age relative to a certain date, in a “on [x] date, you were/will-be [y] years old” manner. The function below can be used to calculate age relative to any date:

CREATE FUNCTION fn_CalculateAge(@BirthDate datetime, @DeathDate datetime)
RETURNS int
AS
BEGIN

-- if @DeathDate is passed NULL, use today's date
SET @DeathDate = ISNULL(@DeathDate,GETDATE());
DECLARE @Age int;

IF @DeathDate > @BirthDate
BEGIN
    SET @Age = YEAR(@DeathDate) - YEAR(@BirthDate);
    IF (MONTH(@DeathDate) < MONTH(@BirthDate)) OR (MONTH(@DeathDate) = MONTH(@BirthDate) AND DAY(@DeathDate) < DAY(@BirthDate))
        RETURN @Age - 1;
END

-- function will return NULL if the dates are in the wrong order, or if the date-of-birth is NULL
RETURN @Age;

END

Final Words

Don’t dismiss the DATEDIFF function entirely in light of its semantic limitations; it’s still useful as long as you remember that it counts date/time boundaries instead of giving a true “difference” between two dates. When calculating ages, be sure to steer clear of it, however.

mcNetworkCopyI’m pleased to announce that the project page for my Media Center add-on, mcNetworkCopy, is now online:

mcNetworkCopy – A network browser and file copier for Windows Media Center

You can download the installer and full source code from the above location.

This provides the opportunity to see how all of the topics covered in my Media Center software development series (MCML, entry points, UI elements, model items, etc) come together into a finished add-on. I hope you find it useful – I certainly do, when the weekend rolls around and I want to transfer my downloaded TV shows to the media box! As always, comment if you have any feedback, positive or negative.