Realisable

Integrating WMSs or 3PL with your ERP

adrian — Mon, 25 Feb 2013 22:21:35 +0000

Introduction

Integrating any Warehouse Management System (WMS) or Third Party Logistics (3PL) with an ERP product such as Sage300, Sage200 or X3 is a complex task. Consideration needs to be given to a variety of issues such as:

Synchronisation of stock systems
Integration touch points
Volume & frequency

This blog post draws on our experience of integrating a number of WMS systems with ERP products, and while not necessarily offering any definitive answers, hopes to highlight the potential questions and issues affecting most WMS integrations.

Stock system

Unless a WMS has been purpose built to utilise the ERP’s stock system, most WMSs maintain their own stock system, each affected by transactions. The transactional nature of the interaction means the two systems will have natural and unnatural differences brought about by the latency in processing transactions and processing errors.

The integrator’s job is to make the ERP and WMS stock systems as synchronised as possible, or at least highlight the possible points of difference.

Logical partitioning

Segmenting a warehouse (or ERP) into areas such as the receipt dock, quarantine and pick face makes perfect sense in both the WMS and the ERP. There are two key integration questions which need answering:

Mapping: how do the locations in the ERP and WMS map to each? How does this mapping affect the stock position and availability of stock in the ERP?
Stock movements: when stock is moved in the WMS from one logical area to another, does the movement also need to be reflected on the ERP? Typically, ERPs treat a location as a boundary for sales and purchase order purposes, thus a movement from quarantine to the pick face will need to trigger a corresponding movement in the ERP.

Time sensitive

Transactions should be processed in the order they were processed on the WMS. If batching groups of transactions for processing, an alternative is to process all the transactions that add stock to the ERP system before those that decrement stock which limits the scenario where a transaction removes stock required by another operation.
Transactions such as P/O receipts can affect the available to sell value in your ERP, and thus affecting potential sales.

Decide the integration points

The obvious candidates for integration are:

Item Master download to the WMS.
Sales Order download, followed by the upload of picking information creating a shipment/invoice in the ERP. Consideration should also be given to the way back orders will be handled.
Purchase Order download, followed by upload of receipt confirmation creating a purchase receipt in the ERP.
Adjustment upload from the WMS to the ERP. Adjustments can take several from write-offs of damaged or end-of-life stock to adjustments made through cycle counts. Be prepared for some heavy debate around the ‘should cycle count adjustments be immediately uploaded/treated as gospel’ question. (They should, but some warehouse managers/business owners seem to have other ideas).

Less obvious or more exotic transactions need some thought:

Transfers: if there are multiple warehouse(s) or the warehouse is partitioned (see logical locations above) a corresponding transfer in the ERP solution will need to be performed.
Customer and supplier returns: typically these are handled manually as the volume and workflow complexity usually means the cost for integration outweighs the benefit.
Kitting: kitting, or light manufacturing, can involve workflow, and the frequency may be too low to warrant integration.
Packaging operations: anything to do with packaging is usually met with a degree of complexity. The purchasing process involves a PO, receiving is done typically at the warehouse door, but thereafter it becomes complex: the receipted goods are consumables, so they cannot be booked into stock, and because of this stock control and management can become more difficult.

Locking and recall

Consideration may be given to both the workflow of the WMS and ERP products. If for example a sales order starts to be picked in the WMS, does the change in status also need to affect the ERP preventing it from being modified?

Similar considerations need to be given to modifications made to, or deletions of sales and purchase orders in the ERP.

Inventory transactions

Inventory transactions by their very nature are more difficult to handle.

Stock is sensitive: invalid stock levels can trigger exceptions, stock costs particularly for adjustments need to be considered.
Shipments and receipt transactions need to be mapped back to their original order documents. That is, when a line is picked, the same line on the sales order needs to be affected.

Volume

Due to the volume and different types of the transactions, the integration must work! Attempting to unpick issues when things go wrong is time consuming, costly and sometimes impossible.

Batch and serial numbers

Batch and serial number integration adds complexity as they increase the number of reconciliation points between the WMS and the ERP solution, where each batch or serial number in the WMS must also be present in the ERP. For example, a warehouse managing 200 batch controlled items may be made of a 1000 discrete batches.

There are some variants on batch and serial number processing which can reduce complexity:

Record batches and serials only on the WMS.
Record only batches and serials only on some operations i.e. picking or goods out.

Methods to integrate

As described in a previous post, we recommend staging tables for data exchange. However, when integrating a 3PL, your method for exchange will be file exchange via FTP/SSH.

When using a staging table, you have the option of using a single table for all transactions or a table for each transaction type. A single table does introduce some degree of abstraction (each field needs to be named FIELD1, FIELD2, etc. which is then logically mapped depending on the transaction type), but it has the major advantage that it can be reconciled more easily and transactions can be easily ordered by date/time.

Reconciliation

The ability to reconcile the ERP and WMS stock systems is paramount. We would recommend creating a report which can be run daily, at a time when no-one is using either the WMS or the ERP, comparing the WMS & ERP stock levels.

Test, test, test

It goes without saying that all integrations should be tested, but WMS integrations more so. The number of transactions, the time sensitive nature, and the variety of transactions makes it a difficult beast. Try to cover the following when testing:

Performance: try to mimic the load and volume of transactions performed throughout the day.
Concurrency and multi-user environment: test for scenarios where transactions may be locked by the ERP or the WMS.
Check your audit points and controls: if you need to unpick an error, ensure you’re capturing all the necessary data from the outset.

Conclusion

Integrating any WMS is not simple. Consideration needs to be given to a wide variety of issues and integration points where cost is often a factor. With careful planning however, it can be very rewarding: seeing all the transactions tick through is quite nice!

Effective use of RVSpy in Sage300 development

adrian — Tue, 19 Feb 2013 15:43:28 +0000

Introduction

Last week there was a post on the Sage300/Accpac VBA forum asking how to automate the Reversals function in Bank Services. In short the responses were: macro recording didn’t work; the view was clumsy and difficult; and the best method to automate would be to mimic the entry via SendKeys.

Given that macro recording is not supported by the bank reversals function, it is necessary to use RVSpy to understand the view calls. This post will describe the various tips and tricks we use when we have to use RVSpy, using the reversals example as per the forum post.

A copy of the final source code and the RVSpy log can be found at the bottom of the post.

Filter the view calls

By default RVSPY captures all calls which leads to an extremely verbose and sometimes incomprehensible log file. Filtering the view calls to remove all the metadata and get requests will significantly trim the log file to the calls which are most important.

Filter view calls: Fields, Field, Attribs, Type, Get, Name, Presents, FieldName & FieldExists
Turn off ‘Show Nesting’: Typically you are only interested in calls the UI makes, and not calls views made to other views.

In our example the size of the generated log went from 1.8MB unfiltered to 427KB filtered.

Recording the entry

Finder Fields

Try to enter the values into each field. Using the finder will significantly add to the log and can make it difficult to understand (see below, for exceptions).

Use comments

Before you do something major such as posting, or adding an entry, insert a comment into the log. It will help you later on to identify the event in the log file and will also provide you with a cutoff where major processing is performed by the view.

The oogily-boogily nature of the Bank Reversals View

If you look at the Bank Reversals view, you can easily understand the trepidation as it appears to have several purposes:

As a history, where posted reversals can be queried and retrieved.
The screen supports the ability to select multiple documents to reverse, so there are a range of fields to support the selection process.
Posting of reversals where there are several fields with conflicting names such as BANK/PBANK and SRCEAPP/PSRCEAPP.

The other major issue is around the view protocol: ‘Flat, Ordered Revisions’.

Flat Protocol: Flat is simply a list or table of records with no hierarchical element; a lot of the setup views such as Account Sets are flat.
Ordered Revision: Ordered Revisions means that an internal list of changes are maintained in memory until a Process or Post command is given. In our example an Insert is performed prior to the Process. If processing multiple reversals, each reversal would be Inserted before the Process performs the reversals as a batch operation.

Understanding Your Log File

After recording is complete, you need to now understand the log file.

In our example the two major issues were the fields we needed to set, and the calls required to post the reversal.

To understand the fields which need to be set, search for ‘Put’. In our example the first put is to PBANK to set the bank code. One thing to note is that RVSpy uses field numbers as opposed to names, so you will need to translate these into names using the Accpac Object Model.

Continue searching and you will find Puts to PAYORID, TRANDATE, and REASON fields.

Next you will find Puts to two key fields SERIAL and LINENO.

These two fields were a little confusing at first but we know they had to do with the transaction being reversed. In both recordings we made we actually used the finder to select the transaction we were to reverse. Prior to the two Put statements we could see the finder searching BKTRANH & BKTRAND, thus the two values corresponded to the keys in these tables.

The final step was to search for the comment we made before we posted the reversal.

Three actions were performed:

Inserting the record into the view.
Switching the view mode to ‘Process Reversal Selections’.
Issuing Process to process the reversal.

Finally

It may be necessary to record something more than once. You may enter something incorrectly, or you may need to repeat an action to understand each call; but with effective use, RVSpy is the best or only tool to understand those Sage300 views/screens which do not support macro recording.

Download Bank Services Reversals source code

Creating prepayments for O/E transactions in Sage300

adrian — Thu, 31 Jan 2013 16:02:52 +0000

This blog uncovers the mystery of how to create A/R prepayments that are associated to their O/E documents and can be accessed via the drill down button on the A/R receipt entry screen. The logic described and presented in the blog applies equally to O/E orders, shipments and invoices.

Creating a prepayment with an O/E document is difficult for a few reasons:

Macro recording does not work; you need to use RVSPY to understand view calls.
Creating a prepayment requires manipulation of three separate, un-composed views and due to the lack of composition each view requires each field to be set individually.

There are three rough steps to creating a prepayment:

Create an A/R receipt batch
Create an O/E prepayment record.
Update the O/E transaction header view.

Full source code is available for download at the bottom of this post.

A/R Receipt Batch

Firstly, it is necessary to either create a new A/R receipt batch or use a currently open batch. Creating the batch is relatively straight forward and really only requires you to set the bank account. Note that you do not need to compose each of the receipt views since you are not creating any receipt entries directly.


 
'Open the A/R Receipt Batch View
Dim ARReceiptBatch As AccpacView
mDBLinkCmpRW.OpenView "AR0041", ARReceiptBatch
 
'Create a new batch
ARReceiptBatch.Fields("CODEPYMTYP").PutWithoutVerification "CA"
ARReceiptBatch.Fields("CNTBTCH").PutWithoutVerification 0
ARReceiptBatch.Init
 
'Set the bank to which the receipt will be entered.
ARReceiptBatch.Fields("IDBANK").PutWithoutVerification sBank
 
'Set the bank currency...if you weren't setting this and were using the default
'bank account currency you can obtain it from the view.
If sBankCur <> vbNullString Then
ARReceiptBatch.Fields("CODECURN").Value = sBankCur
Else
sBankCur = ARReceiptBatch.Fields("CODECURN").Value
End If
 
'Set a batch description
ARReceiptBatch.Fields("BATCHDESC").Value = "Prepayment Batch from O/E"
'Update the batch
ARReceiptBatch.Update

Update the O/E Prepayment View

The second step is to create the O/E prepayment record. The prepayment record is responsible for creating A/R receipt entry and it also associates the receipt to the O/E order so that there is a drill down against the prepayment.

Each O/E transaction (Order, Shipment & Invoice) has a separate prepayment view where each is identical with the exception of the view id. Therefore the logic used here is applicable to each.

If you are familiar with Sage300 development, most of the views are nicely programmed where a lot of hard work is performed behind the scenes by the view. Unfortunately, the prepayment views are the complete opposite! They do not have any default values either from the customer or order, so it is necessary to explicitly set each field.

The code sample below illustrates this, where it’s necessary to set the Customer, Customer Name & Customer Currency all individually despite setting the Customer Id.


 
Dim OrdPrepayments As AccpacView
mDBLinkCmpRW.OpenView "OE0530", OrdPrepayments
 
OrdPrepayments.Init
'Associate the prepayment with order
OrdPrepayments.Fields("ORDUNIQ") = OEOrdHeaderFields("ORDUNIQ").Value
OrdPrepayments.Fields("CUSTOMER").Value = OEOrdHeaderFields("CUSTOMER").Value
OrdPrepayments.Fields("CUSTDESC").Value = OEOrdHeaderFields("BILNAME").Value
OrdPrepayments.Fields("CUSTCURN").Value = OEOrdHeaderFields("ORSOURCURR").Value

Update O/E Header View

The final step is to update the prepayment fields on the O/E transaction header view, which update the Total Prepayments field on the order’s Totals Tab.

The attached code is a full working sample illustrating how to create and attach a prepayment and should work for Sage300 versions 5.6 and above.

Download Sage 300 OE Prepayment source code

Dreaming for errors – integrations, error handling & logging

adrian — Fri, 25 Jan 2013 22:27:00 +0000

Don’t think that all of your data is going to import every time, if you’re lucky some of it might some of the time.

When designing or writing your integration application it is essential that you expect errors and, as a result, design your program with the possibility for errors at the forefront of what you’re trying to achieve. In fact errors are probably just as critical as the data itself, if not more so.

This blog describes our best practices for designing error handling and logging within an integration.

Design Pattern – The Layered Approach to Error Handling

In a previous position I had the unenviable task of having to maintain several of a colleague’s poorly written import programs. Each time I made a modification I would re-factor a certain part of the program just for my own sanity and in conjunction I maintained several integrations I’d written myself. Over time each import converged to a similar structure, all based upon the way the integration handled errors.

The resulting pattern was a 3, sometimes 4, layer structure where each layer had its own error handling responsibilities.

Atomic transaction: This is the inner most layer of your integration and is responsible for managing a single ‘atomic’ transaction. On failure it is this layer which cancels or rolls-back any half written transaction.
Transaction handler: The transaction handler is essentially a loop which calls into the ‘atomic’ transaction handler. When the atomic handler raises an error, this layer needs to decide how to proceed: whether to move to the next transaction or abort.
Task handler (optional): When your integration interfaces with anything more than a single end-point the task handler manages the handling of exceptional and non-exceptional data. Where a transaction succeeds, it needs to be submitted to the next end-point, whereas a failure usually means that it should not be processed at the next.
Entry point: The outermost layer to your application. It is responsible for:
- Generation of the end user log or processing report.
- Catching and logging any unexpected or catastrophic errors.

Atomic transactions

It goes without saying that whenever an error occurs in your automated integration it shouldn’t leave half written, partial transactions, database records, or files.

Sometimes however the concept of a transaction needs some thought, and requires input from business requirements, the technology of the platform and the ability to re-process data.

For example, should a batch of cash transactions into an accounting system be imported in an all-or-nothing approach, or should the integration import the ones it can, omitting the erroneous transactions? This scenario has implications on where the batch is posted; without the erroneous transactions, it may affect the bank reconciliation function at a later point.

Logging types

In our experience there are two audiences for the logs of any integration: end-users and technical staff. Users will want something they can understand, and technical people want something they can use to debug or understand a problem.

End-user reports

Are sent to the people responsible for monitoring any integration/data feed.
Should be in readable, easily digested format.
Should allow the user to be able to quickly reconcile data passed between applications.
Should provide the user enough information to allow them to identify and/or resolve the issue, or at least identify any erroneous data.

Catch all & Catastrophic error logging

Used to catch unexpected errors and exceptions, useful for technical staff to help identify isolated issues.
Catastrophic errors are simply those you didn’t plan; for example, a loss of network connection; a corrupted file; a sudden disconnection from a database; an untimely permissions change; or out of memory exception.
In contrast, an invalid item code on a sales order import is not catastrophic; it should be handled by your standard logging mechanism and reported on your end-user reports.
These errors should be logged locally to either a local file or local database, windows event log. Don’t rely on remote resources such as remote databases or email as network availability could be limited or the cause.

Runtime logging

Errors, warnings & messages should be persisted as early as possible, which means writing to a database or file as and when they occur. Should a catastrophic error occur you have a process log, a trace, which would otherwise be lost if they were held in memory and persisted at the end of the process.

Log libraries

It is worth mentioning that there are a number of open source logging and tracing libraries available such as nlog, log4net, the .net trace libraries and Microsoft’s Enterprise framework which can provide you with a robust and feature rich way for persisting errors and warnings to a number of mediums. If you decide to use these, as opposed to rolling-your-own log scheme, the libraries can require some understanding to configure & setup and their outputs are not suited for end users.

Generating the end user process report

At the conclusion of processing it is necessary to generate the process report for the end users. If you’ve followed our recommendations, this should mean re-querying the medium to which the errors were persisted during processing and re-formatting them into an easy to read format.

The best and easiest format to send the end users a process report is email for the following reasons:

Everyone has email.
Portable and amendable: Annotations can be made and forwarded to others if remedial action needs to be made by another person.
Logs can be easily formatted with html and presented nicely.

Summary

Error handling and logging is a key component of any application, more so with integration! Without proper error logging and reporting, your organisation will at best spend needless time resolving and reconciling data between applications and at worst lose customers because of poor service and lost transactions.

We hope this has been of use to some of you and if you ask us nicely, we may just provide you with some samples and template code illustrating the best practices we’ve described.

Printing and multi-thread issues with Sage200

appiah — Thu, 17 Jan 2013 09:00:01 +0000

In this blog we take a deep dive into the mechanics of using the Sage reporting engine in an external application using the Sage200 SDK for both single and multi-threaded applications. Topics covered:

Correctly referencing the Sage reporting engine
Using the Sage reporting engine within a service
Use the Sage reporting engine in a multi-threaded application
The Sage200 Security Context
Multi-threaded use of the Sage200 Application object

Full source code is available at the bottom of the blog.

Background

When we added to IMan functionality to print various Sage200 forms we encountered a number of issues during development, deployment and use in a production environment.

This blog will attempt to uncover and present the solution to these issues. We hope that this will help other Sage developers who are grappling with the reporting engine complexities.

External printing with Sage Report Engine

The first phase to adding print support to our application was to provide support for printing forms such as invoices, picking lists, etc.

The biggest hurdle we faced was the correct referencing of various Sage libraries and dependencies.

References

Copy \Program Files\Sage200\ Unity.config to your local bin directory.
Reference the following assemblies and ensure that NONE are copied to your local bin directory.
- Sage.Reporting.Services
- Sage.Reporting.Model
- Sage.Reporting.Engine.Integration
- Sage.Query.Engine.Model
- Sage.Query.Engine

Setting the Printer at Runtime

At the time of writing (an enhancement request has been logged with Sage) the Sage Reporting Engine does not have any facility to set the printer at runtime. It can however be set by traversing the internal object structure of the ReportEngine object.


/// 
/// Prints a report to a specific printer or exports a report to a file.
/// 
/// The file to export the report to, can be empty or
/// null if the report is being printed.
/// The IExportService i.e. this object.
/// Should always be set to SuppressUserInteraction.
/// The type of file if the report is being exported or
/// prn if being printed.
/// The local printer name if the report is being
/// printed.
/// True...not sure of the signifigance as it's undocumented.
private bool PrintReport(string exportFilePath, IExportService exportService, ExportFlags flags, ExportType exportType, string printerName)
{
  Type exporterType = exportService.GetType();
  PropertyInfo reportsPropertyInfo = exporterType.GetProperty("Reports", System.Reflection.BindingFlags.Public | System.Reflection.BindingFlags.Instance);
  IList reports = reportsPropertyInfo.GetValue(exportService, null) as IList;
  // Should never be zero, but just in case.
  if (reports.Count == 0)
  {
    throw new Exception("No reports in the layout file");
  }
  // Get the ReportBatchItem
  var batchItem = reports[0];
  // Get the report.
  var reportPropertyInfo = batchItem.GetType().GetProperty("Report");
  // Get the actual report object.
  Sage.Reporting.Model.IReport report = reportPropertyInfo.GetValue(batchItem, null) as Sage.Reporting.Model.IReport;
 
  // Set the printer.
  // Sometimes the PrinterSettings can be null (usually when the report is
  // saved with a printer which is not installed/available under the user
  // profile which the report is being run.
  if (report.PaperLayout.PrinterSettings == null)
  {
    System.Drawing.Printing.PrinterSettings printSettings = new System.Drawing.Printing.PrinterSettings();
    // Set the printer name before setting the object to the PrinterSettings
    // property. 
    // Setting the object to the property fails, with the property remaining
    // null.
    printSettings.PrinterName = printerName;
    report.PaperLayout.PrinterSettings = printSettings;
  }
  else
  {
    System.Drawing.Printing.PrinterSettings printerSettings =  report.PaperLayout.PrinterSettings;
    printerSettings.PrinterName = printerName;
  }
}

Using the Sage Reporting Engine within a service

Using any library within a windows service deserves special attention due to the various restrictions of the running environment, the Sage reporting engine is no different.

ExportFlags.SuppressUserInteraction: Everything in a service needs to run attended therefore it is necessary to instruct the reporting engine not to display prompts or message boxes. Further, service security in Vista and Server2008 was tightened so that any service attempting to display any UI element is terminated by Windows.
Sage.Accounting.DataModel.dll : Copy Sage.Accounting.DataModel.dll located in C:\Users\\AppData\Local\Sage\Sage200\AssemblyCache to your service’s bin directory. As the name suggests, the Sage.Accounting.DataModel.dll assembly contains the Sage200 data model that is dynamically built by the Sage server whenever a model extension is installed (for instance when you add a new view or table to the database and run through the Custom Model Builder).
Use locally installed printers: Services don’t load user profiles (irrespective of the service’s user); therefore they don’t have access to network printers (which are loaded dynamically as part of the user profile).

Using the Sage Reporting Engine in a multi-threaded application

This section deals with the threading issues experienced when our application attempted to run multiple instances of the Sage reporting engine concurrently.

Having completed the initial report printing development we deployed this to a couple of sites which were running without issue.

Our problem came on a new site where each time our application attempted to print Sage200 forms concurrently we would encounter a range of exceptions. These exceptions would essentially terminate the execution of one of the concurrently running integrations.

Background

Our IMan application is multi-threaded, meaning that each time integration runs within a separate thread of a single multi-threaded apartment (MTA) executable.

To this point we had not experienced any threading related problems with IMan. Multiple integrations could be run concurrently without issue. Further, we could successfully run multiple integrations or threads concurrently where each could create transactions within Sage200.

Problem

Under the hood the first thing Sage does when creating a ReportingEngine object is make a call to DoEvents (this infers the Sage reporting engine is written in VB.Net).

Because this ultimately makes a call to the Windows messaging queue, any interactions with the UI or Messaging queue should come from a single thread (marked as a Single Threaded Apartment (STA)) from within an application.

Windows does not permit multiple threads within an application to access the messaging queue, with an exception being raised by windows where multiple threads attempt to do so.

Why is an STA Thread Required? (Simplified)
http://blogs.msdn.com/b/jfoscoding/archive/2005/04/07/406341.aspx

A fuller discussion of COM and threading models.
http://msdn.microsoft.com/en-us/library/ms693344(VS.85).aspx

The error below is the one generated when multiple threads of the same application attempt to create ReportingEngine objects at the same time.

The error is pretty misleading, and at the time we thought that we were dealing with a linking or referencing issue with some Office components.

Given that we hadn’t had a threading issue previously and that we had suppressed user interaction (above), we were left scratching our heads for a while.

The clue came when we looked at the callstack more closely: DoEvents and FPushMessageLoop both indicate some UI interaction is occurring.

System.InvalidCastException: Unable to cast COM object of type ‘System.__ComObject’ to interface type ‘IMsoComponentManager’. This operation failed because the QueryInterface call on the COM component for the interface with IID ‘{000C0601-0000-0000-C000-000000000046}’ failed due to the following error: No such interface supported (Exception from HRESULT: 0×80004002 (E_NOINTERFACE)).
at System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32 dwComponentID, Int32 uReason, Int32 pvLoopData)
at System.Windows.Forms.ComponentManagerProxy.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32 dwComponentID, Int32 reason, Int32 pvLoopData)
at System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context)
at System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context)
at System.Windows.Forms.Application.DoEvents()
at Sage.Reporting.Engine.Integration.ReportingEngine..ctor()
at Realisable.Connectors.Sage200.ReportExporter..ctor(String reportFileName, Dictionary`2 criteria, String exportFilePath, Boolean print, String printerName)

Our requirements

Retain the multi-threadedness of Iman: Changing our architecture to single threaded was a non-starter.
In-process: We did consider writing a separate print spool application, but this suffered from two problems:
- Marshalling the print requests and response between two applications was considered too fragile and difficult.
- Sage200 permits only a single login per user at a time. A separate process would have required a second login and Sage user.

Solution

The solution is a variant of the synchronised singleton pattern (this is in Java, but the pattern applies), where a single thread (print thread) processes report requests singularly, one at a time. Each subsequent print request is blocked until the print thread completes the executing print request.

Implementation

ReportQueueManager

The ReportQueueManager marshals the print requests to the Sage reporting engine and is also responsible for printer thread running state. It contains 4 methods:

RunReport: Called by the worker threads to process a print request.
ProcessReport: The internal method which de-queues and processes each request.
Start & Shutdown: Respectively manage the startup and shutdown of the printer thread.

Printer thread

Internal to the ReportQueueManager class is a single threaded apartment (STA) thread which is responsible for processing each print request.

Inter-thread communication/synchronisation

To communicate between the calling/worker thread and the printer thread are two blocking queues:

Request Queue: Accepts print requests from the worker thread.
Reply Queue: Returns the success/failure of a print request from the printer thread back to the calling/worker thread.

The request queue is the key component for synchronising calls between the worker and printer threads. The queue is set to permit only a single request at a time i.e. its maximum length is set to one. When the queue is full (it has a request), all further requests are blocked; i.e. their execution is halted, thus permitting only a single request to be processed at a time.

To achieve the blocking queue we used a pattern as described here.

ReportRequest

The ReportRequest object encapsulates all the criteria such as report name, report criteria/parameters, printer name, etc. for a print job.

A ReportRequest object is created by a worker thread for each print request. The worker thread calls ReportQueueManager.RunReport, if the queue is empty, the request is processed immediately. However, if the queue is not empty (meaning a report is being printed), the calling thread is blocked.

Communicating success/failure to the caller

To communicate back to the calling thread if the print operation has succeeded or failed, a ReportReply object is created and placed onto the reply queue.

Execution is returned when the ReportQueueManager de-queues the reply object from the reply queue. The reply is checked for success or failure, and where there is failure, it re-throws the exception.

Understanding the Sage200 logon context

Our solution meant that each print request could be run against different Sage companies/databases, so a critical component in our solution was to get ReportEngine to target the correct Sage company.

If you have ever dealt with the Sage reporting engine, you would have noticed that there are no properties or methods for authentication or for targeting a specific company/database. This is because the reporting engine uses the current thread’s security context/principal (System.Threading.Thread.CurrentPrincipal).

Per MSDN, a principal object represents the security context of the user on whose behalf the code is running, including that user’s identity (IIdentity) and any roles to which they belong.

Sage200 sets the CurrentPrincipal as part of the set operation of the Application.ActiveCompany property.


// Authenticate to Sage200
_application.Connect("admin", "admin"); 
// Set the active company.
for (int i = 0; i < _application.Companies.Count; i++)
{
  if (_application.Companies[i].Name == "Sage200_2010_DemoData")
  {
    // When the ActiveCompany property is set Sage sets the 
    // current thread's CurrentPrincipal property with logon
    // context.
    _application.ActiveCompany = _application.Companies[i];
    break;
  }
}

The solution requires that the print thread’s CurrentPrincipal is set to the principal of the thread.

Since the reporting and print threads are separate it is necessary to pass the print thread to the worker’s thread’s principal for each print request.


{
  public ReportRequest()
  {
    //Capture the Principal of the calling thread
    Principal = Thread.CurrentPrincipal;
    //Used for debug and tracing purposes.
    ThreadId = Thread.CurrentThread.ManagedThreadId;
  }
....

We do this by capturing the principal in the constructor of the ReportRequest class. When the ReportQueueManager de-queues the request it sets the printer thread’s CurrentPrincipal to the Principal of the request object.


/// 
/// Worker method to process ReportRequests
/// 
private static void ProcessReport()
{
  // Run forever!
  while (true)
  {
    // This method will block until a request is placed onto the queue.
    ReportRequest request = _requestQueue.Dequeue();
    // Set this thread's principal to the principal of the request, 
    // which the reporting engine then uses.
    Thread.CurrentPrincipal = request.Principal;

The result allows each calling/worker thread to target a separate Sage200 company e.g. worker thread ‘A’ targets company ‘Y’ and worker thread ‘B’, company Z.

It is worth noting that without this the Sage ReportingEngine object will raise an exception when CurrentPrincipal is not set (ref 1).

Sage.Accounting.Application objects are not threadsafe

During testing we kept encountering a DuplicateKeyFound exception when setting the ActiveCompany property whenever two threads entered the method simultaneously. Our solution was to wrap the code for the connect and disconnect in a lock statement.


// The ActiveCompany property setter is not threadsafe, 
// wrap the code around a lock to prevent 'DuplicateKeyFound' exception.
// _syncLock is a static object declared at class level.
lock (_syncLock)
{
  // Authenticate to Sage200
  _application.Connect("admin", "admin"); 
  // Set the active company.
  for (int i = 0; i < _application.Companies.Count; i++)
  {
    if (_application.Companies[i].Name == "Sage200_2010_DemoData")
    {
      // When the ActiveCompany property is set Sage sets the 
      // current thread's CurrentPrincipal property with logon
      // context.
      _application.ActiveCompany = _application.Companies[i];
      break;
    }
  }

Summary

We would like to thank the Sage200 development support for all their assistance, and ESPI and the client for their patience.

Download Sage200 Report Manager source code

(Ref. 1) This is not entirely true! We found during testing that if we had Sage200 open and did not set the printer thread’s current principal the ReportEngine would somehow use the SAAPrincipal of the open application. An exception was thrown when Sage200 was not open.

Integration and Synchronisation Data Exchange

appiah — Fri, 11 Jan 2013 10:00:31 +0000

In this final part of our multi-part post (see part 1 and part 2), I will discuss the various transports which can be used to synchronise data between applications.

As opposed to the previous two parts, the strategies in this post are not limited to synchronisation and apply equally to integration.

File transfer

The old school method for data exchange has existed for as long as computers have interchanged data. File transfer involves one application creating a file and putting it in an agreed location, whilst the receiving application reads, processes and then does something with the file.

Pros

Secure: File transfer can be easily locked down, making it a relatively secure means to exchange data. FTP and SSH are both solid protocols meaning remote exchange over the internet is secure.

Cons

Difficult to audit: Compared with other mechanisms files typically don’t have any natural means to audit or to trace audit.
Remote servers require an FTP/SSH solution to exchange files, adding another chink in the chain of events.
Fragile: Easy to get out of sync where a failure can require significant manual intervention to re-establish normal operation.
CSV and other column based formats such as fixed width text and Excel suffer from brittleness, where file schema modifications (field additions and deletions) require significant re-configuration on the receiving end.

Tips for use

One transaction per file: It’s much easier to identify a particular transaction, and re-processing is easy should something go awry.
For CSV and other column based formats, always add new fields to the end of the file; never remove existing ones, better still don’t use these formats!
Use a resilient API based data format such as Xml or JSON. Do not go overboard however.
Try to avoid Xml schema where possible. We feel schemas add unnecessary constraints to the format, where a substantial amount of the inherent flexibility of the format is lost. They add a level of complexity not required in the SME space.
Always use a recognised API/library when consuming and producing Xml/JSON data. This will at minimum ensure the format is standard compliant. We’ve seen several instances where organisations (that should know better) provide us with improperly escaped Xml, simply because they weren’t using an API to produce the Xml.
After files are processed move them to an archive directory; don’t delete.

Database & staging tables

Our preferred means for both integration and synchronisation, the use of databases & staging tables providing a much more robust transport as compared to file exchange.

Staging tables are an intermediate set of database tables where the sender records pending data for the recipient(s) to later process.

Pros

SQL: Provides vendor and format independence to query and update a database and table schema(s) changes are usually non-breaking.
Audit and Traceability: SQL/Databases provide an easy means to record and query any key data required for auditing purposes e.g. timestamps, error & warning messages.
Durability and Recoverability: Databases are naturally durable and far less vulnerable to corruption. In the event of a failure recovery can be made quickly assuming a backup is available.

Cons

Security: Where interchange take place over the internet the database needs to be open to the outside world leading to potential attacks. These can be mitigated by restricting the connecting IP addresses, use of complex passwords, amongst others.

Tips for Use

Always have the following fields in a staging table:
- Status: Used to record whether a record is pending or processed. Always include an error status which allows you to identify erroneous transactions quickly. It is recommended that an index is setup on the status field to help speed querying.
- Last Modified Date: A timestamp to record when the last update was made.
- Last Update By: Used to record which application made the last update.
- Reason/error/message: Where there’s an error or a warning generated as part of the integration or synchronisation it should be recorded against the original record. This then gives you a robust audit trail which can be used later to trace problems.
- Foreign Record Id: Where a recipient application generates a transaction or record id (such as an Order, Invoice, or even Customer number) update the staging table with this reference.
Backup frequently: Depending on the volume of data or the location of the database i.e. for high volume sites or where the database is publicly accessible this may be several times a day.
Always update the staging table immediately after writing to the destination application.
Where processing takes a long time, consider setting the status field to an intermediate value to prohibit other integration processes from processing the same data.

Message queue

A message queue is a transport level asynchronous protocol where a sender places a message into a queue until the receiver removes the message to process it. One of the key attributes of a message queue, especially for data synchronisation, is that messages are guaranteed to be delivered in FIFO order.

Message queues have traditionally been limited to larger scale integration and synchronisation and where both sender and receiver exist on the same network. However, with the explosion of cloud computing and an agreed standard (AMQP), there is an increasing prevalence of low cost services:

Windows Azure Service Bus
Amazon SQS
Storm MQ: Currrently in-beta this ‘free’ service looks promising.

Pros

FIFO: Messages are guaranteed to be delivered in order.
Robust and Transactional: Message queues can be backed up, and are naturally transactional.

Cons

Message Queues are a Transport: The data format of the message still needs to be agreed between sender and recipient.

Tips for use

Use a flexible data format such as Xml.
File and database recovery is well known, message queues less so. Understand how to administer your message queue technology and how to recover on the event of a failure.
Prevent message build up by having an application which is constantly listening for incoming messages.
Use different queues for each data flow; don’t try to multiplex multiple message types onto a single queue.

No interchange

In this scenario there is no intermediate data form, instead each application holds its pending data until the recipient application retrieves it.

Unless an application specifically integrates with another this style of interchange requires some form of middleware to mediate the interaction since:

Each application has its own interface i.e. webservice or a COM/Java/.Net API.
The data usually requires some transformation for it to ‘fit’ with the other.

Summary

If a choice can be made, use databases/staging tables as they are by far the easiest and most robust strategy.

We expect an increased use of message queues, especially with hosted infrastructure environments such as Azure and Amazon.

CSV: Dear oh dear! It may be easy to produce and simple to read, interpret and load into Excel. But it’s 2013! And there are more sophisticated and better means for exchanging data, so why are we still using this archaic format?

Synchronisation strategy

appiah — Fri, 30 Nov 2012 09:00:42 +0000

In Part I of this multi-part blog some of the most common issues for synchronisation were discussed.

This second post will describe the strategies available for synchronising data between applications, their advantages and pitfalls.

The strategy is the means to determine when a record should be inserted, updated, or deleted.

Data Structures

It is necessary to first understand the issues surrounding more complex data structures.

Simple single level data structures, such as customer records, can be readily synchronised. However, as data structures become more complex so does the strategy.

The above example is illustrative, both order and order lines must be synchronised; an order without lines is not an order. However, the third level of data – the serial–numbers add several complications:

Is it necessary to synchronise them? If they are used for transactional purposes, yes, but a view-only application may not require them.
What do the serial numbers mean when an order line is updated? Are they a fresh set of serials, or just additions? While this question applies also to lines, the deeper the structure the more pronounced the issue becomes. This is due to the feedback of how the update is made on each level of data.

Means of Identifying Updates

Inserting & Updating Records

Inserts and updates follow a simple pattern where the destination application is queried to identify if it contains an existing record. An update is performed where a match is found; otherwise when no matches are found, an insert is performed.

Deleted Records

Handling deletions is a much harder issue, primarily as the delete needs to be identified. There are three approaches:

Tombstoning

Tombstoning is a technique which instead of actually deleting the record, marks it as deleted, via a field within the record’s table. What makes this a preferred method for handling deletes is that the tombstone makes the delete operation explicit and in turn reduces complexity around synchronisation.

Tombstoning however requires the source application to provide such functionality (usually only a feature when data synchronisation/integration is integral to the application e.g. CRM).

For applications that don’t support tombstoning, it is necessary to record the record deletion(s) in a separate interface table or file, usually by way of a trigger (to be discussed in part 3 of this blog).

Record Comparison

Record comparison involves comparing a set of data from the source application to the set of data in the destination application. A delete is performed when a record is found in the destination but not in the source.

In contrast to ‘Tombstoning’ this suffers from three primary problems:

Complex: Record comparison is implicit; it requires the ‘data synchroniser’ to determine if a record has been deleted.
Auditing: There is a lack of an audit trail where records just disappear, with no facility to check why/when a deletion occurred.
OK for child records not for the primary entity: Using comparison for child records in hierarchical structure, e.g. order lines within an order, can be appropriate where there are limited records and the detection of deleted records is cheap and relatively straight forward.

Using comparison for a primary entity (e.g. the order) requires the examination of a large set of data to determine whether a delete is required.

Re-create Everything

A third alternative is to delete and re-create all records on each ‘sync’. This strategy is easiest to implement, covering inserts, updates and deletes, since it doesn’t require any logic to detect and handle changes. However, its use is only limited to the following scenarios:

Data is loaded at a database level e.g. reporting & data warehouses and CRMs. Applications with a sophisticated data structure and/or API usually frown on this approach as business logic and data validation is side-stepped and can lead to integrity issues.
The destination application is view only and non-transactional.
There are no (or very limited) foreign-key constraints: Foreign keys enforce relationships between records. When a record is deleted so must all the associated records. For example, deleting a customer record also requires deleting all the customer’s associated order & invoice records.
Data size: Deleting and re-inserting all records in a large dataset may take a long time to complete and can be expensive in terms of computing power, particularly with hosted environments where there can be CPU cycle costs. Pruning the dataset, usually by date, is an effective strategy to limit the load size.

Handling Application Logic

When synchronising data between applications it is necessary to negotiate and handle the restrictions or logic the API applies to the data.

Business logic is much stricter than any database enforcement.
Transactional data moves through a process, where the status of the transaction (or line) dictates whether modifications can be made.
Deletes are never easy – Particularly with ERP applications, deletes are handled in a number of ways: master items may need to be marked as inactive (a form of tombstoning); transactions can either be deleted or cancelled.

Strategies for handling application logic

Atomicity: An update should be all or nothing; never have a situation where some updates to a single transaction or master item fail and some succeed. Whilst this should apply whether synchronising at data level or application level, it’s easier to fall into a situation where a partial update is made due to lack of understanding of the API.
Is your synchronisation dumb or smart: You will need to make a choice as to how much logic you embed into your synchronisation, and often they start dumb and evolve to smart. What this means is the amount of logic required to cover certain scenarios or use cases. At the stupid end, there is no logic and a simply try/catch block is placed around your processing to catch any exceptions. Conversely, at the smart end you may check for an entity’s existence, raise workflow events, or handle specific edge cases.

Summary

This post hopefully has helped to explain the most common synchronisation strategies and pitfalls. It is by no means complete and we would like to hear your opinions.

The next post will discuss the various means i.e. file transfer, database, etc. available for data synchronisation.

Common issues when synchronising data between applications

appiah — Fri, 16 Nov 2012 10:00:00 +0000

In this first of a series of posts, I will layout some of the common issues encountered specifically when synchronising data between applications.

Synchronisation is the replication of data between two or more applications. It is usually more than a one-way push. Synchronisation is the continued updating of data to keep all applications reflecting the same view.

Firstly, there are two broad categories, each with its own set of properties, that data falls into:

Master Data

Master Data is conceptualised as key entities used within an organisation. Customers, Employees, Items & Suppliers are all considered master data. At the simple end of synchronising, Master Data records involve a single record of data per entity. However, Master Data often has a number of child attributes such as addresses for customers, bank details for employees, and a range of attributes such as locations, suppliers and units of measure for items.

It is increasingly expected that Master Data be updatable from multiple applications where each application holds its own copy of a common ‘truth’.

Transactional Data

Whilst sometimes more complex due to their header/detail structure transactions, they are usually easier to synchronise due to the following characteristics:

Simple direction/flow – One application is the master and the other(s) are slaves thus eliminating issues such as conflicts.
Transactions are less likely to be updated – Once a transaction is posted it has reached its final state and requires only a single push to the target application. There are however a couple of notable exceptions: Sales & Purchase Orders often need to be synchronised to reflect their current ‘state’ and because of their more complex structure and heavy business logic require careful attention. Further, Customer & Supplier Payments can have a complex many-to-many interaction with their corresponding invoice documents.

There is a third category: Master Data with transactional properties. An example of this is a project structure, where the project goes through several stages throughout its life and where each stage has different business rules affecting its behaviour.

Issues Surrounding Synchronisation

Data Structure Complexity – More complex data involving header, details, sub-details, etc need strategies which address changes for each ‘level’ of data.

Synchronisation & Conflict Resolution

Synchronisation needs to be considered at each level of data.
Deciding which application, if any, is the master or whether bi-directional integration will be supported.
Conflict Resolution – What rules should be followed where there is a conflict and is there sufficient data available to resolve a conflict and/or data storage to mediate conflicts?

Data Impedance & Mismatch – The higher the resistance or more transformation required the less able you are to synchronise it.

Field Level Transformation – Destructive transformation, where the result of a derived or calculated field cannot be applied in reverse or accurately reversed can prevent bi-directional synchronisation.
Field Lengths and Type Differences – Two common issues arise: truncation of textual fields such as name, addresses & comments, and the mapping of enumerated fields i.e. drop down fields, where the choice is identified as an integer value within the source or target application.
Data Structure Mapping – Cases where the structure of the source data does not match the target and where transformation is required to massage the data into the required shape.

Data Interchange – As the sophistication, complexity and/or frequency of the synchronisation increases so does the requirement for more sophisticated means for transferring data.

Data Take On – Finally, issues around the amount of historical data, whether the load time for data is significant and whether data cleansing for old or historical data needs to take place.

In subsequent posts I will describe the various strategies available to resolve these issues.

Achieving the original aim?

appiah — Fri, 02 Nov 2012 10:00:20 +0000

If you’re working with a single product day-in day-out for three and a half years you can sometimes lose sight of what you’re trying to achieve: tunnel vision! My aim in creating IMan is to create a general purpose integration platform with a strong focus on usability and one which is flexible enough to cover 95% of all integration requirements.

Have these been achieved? I would like to share some insights on the development process, the goals, and some of the problems I encountered along the way.

Five favourite parts of IMan

The Preview

The preview grid, situated at the right half of each transform setup interface, is populated with the results of the transformation each time the ‘Refresh’ button is pressed. It is one of the key pieces to facilitate easy and rapid integration design and, as one user put it, with the preview area ‘you’re not working blind’. Development, however, was not easy! Several weeks were spent, several calls made to Microsoft developer support, and many grey hairs grew! The end product involves a multi-threaded WCF windows service with out-of-process COM Interop (I learnt a lot)! The end result is that multiple designers can be working simultaneously on a single instance of IMan.

Common Connector Interface

The same interface is used for mapping Sage300 as Sage200 as SageCRM. This has been an evolutionary development, but it results in a common interface for every application, especially helping those consultants working with multiple products.

Hierarchical Support & Associated Transforms:

There was no point in producing another integration product where hierarchical data support was shoe-horned, so IMan has been built from the ground up to support hierarchical data. The goal is to make the design process as intuitive as possible, whilst increasing the software’s flexibility.

Free Form Designer

To achieve the ‘95% integration coverage’ IMan is designed so that each transform/node within the integration can be linked to the previous transform ad infinitum, permitting multiple transforms to be linked or branched to build complex integration logic.

The major challenge was to provide a design surface which allowed interactive diagram creation similar to that of a flowchart i.e. to visualise the underlying implementation. Also, it needed to be in a web environment.

To achieve this, the only real choice was to use an off-the-shelf diagramming component (not an easy task as it needed to conform to web 2.0; it was 3.5 years ago). In-house development of such a component would have been sheer lunacy.

The result is the design surface we have today. We believe it facilitates everything from simple dataflows through to more complex integrations. Further, due to the flexibility, we’ve seen several of our customers use IMan for complex process automation, far beyond my original intention of IMan as a general integration tool.

Drop In Connectors & SDK

As a general integration tool, IMan required a means to connect other applications through their APIs, particularly with the marked increase of hosted applications where a webservice connection is the only choice. The initial thought was to create a generic webservice connector, but the different protocols, implementation and necessary logic, plus a UI to configure it was just too complex (just look at SData for example), and out-of-step with IMan’s focus on usability.

To facilitate this, we’ve taken a low level approach and added the ability for third parties to develop their own connectors for external applications, web services, etc.

I believe we’ve taken the correct approach as IMan gives you the ability to connect to any application, not just webservices; further, it enables the developer to add ‘smarts’ into any connector.

From a developer’s perspective, I think the thought of connector add-ins is awesome!

Our partner, EC Internet, has already developed the Magento connector, with two more on the way for Vortex ASP.Net Storefront eCommerce platform and Sage Pro.

Least Favourite

It’s only fair to share this, so here’s the hall of shame! Needless to say each of the points below are on the enhancement list:

The Diagram Component

It’s not that bad, but it can be slow, clunky, and connectors can be difficult to manipulate, especially on large graphs.

Setting Up Hierarchical Datasources

Field names aren’t inferred for file based datasources, so you need to enter them manually, and the screen handling could be better!

Setup Tab Load

Why does it take so long to click on the setup tab?

Load Time

Not our fault, but I am pretty sure we could do better!

We’d like to know your thoughts: have we achieved our design goals? Are there other clunky features which need improving?

Main Page Image 5

James — Mon, 12 Oct 2009 10:27:57 +0000