File Events¶
The scheduler service uses file events to watch for specific file activity and run integration jobs when a specified file event occurs.
IMan can watch activity on the IMan server's own file system, on a remote path or on a cloud file system: an AWS S3 bucket, an Azure Blob container, or a OneDrive or SharePoint drive.
Example
A file event can watch a particular directory on a local or remote server for any new files. When a file is created in the directory, IMan triggers an integration that uses the new files as input to the reader transform.
Setup > File Events¶
Delay on Events¶
IMan ignores any event raised on a File Event record within one second of another.
On a local or remote Windows path, this smooths over Windows raising several events for the same action. The same one-second window applies to a cloud file system, where it merges the burst of notifications that a single upload can produce.
Adding/Modifying/Deleting Events¶
After you add, modify or delete an event, restart the Realisable Scheduler Service for the change to take effect.
User Permissions¶
Integrations triggered by a File Event run under the same security context as the Realisable Scheduler Service user.
A File Event on a cloud file system is the exception: it connects with the credentials held on the file system record, not with the Scheduler Service user.
File Event Setup¶
You set up the file activity monitor on the File Events screen, on the Setup tab of the primary navigation strip.
ID¶
The Id of the File Event.
Description¶
The description of the File Event.
Integration Id¶
The job to trigger when the event condition is met.
File System¶
The file system to watch.
Leave it as Default to watch the IMan server's own file system. Select one of the file systems defined under File Services to watch cloud storage instead. See Watching a cloud file system below.
Path¶
The path to monitor. It can be a local or a remote path.
For a remote path, specify a UNC path. Avoid mapped drives, which are fragile and cause complications when the scheduler runs as a service.
Where you have selected a file system, the path is relative to that file system's root: the bucket, the container or the drive. You do not need a leading separator.
Filter¶
The files that trigger the event. The filter may use the wildcards ‘*’ and ‘?’.
Example
*.csv¶
- Matches files with a ‘.csv’ extension.
20??_??_??_Import.xml¶
- Matches the following: 2010_01_23_Import.xml and 2000_10_23_Import.xml but does not match 1999_12_11_Import.xml or 2010_02_23_Import2.xml
*¶
- Matches all files.
Watch Events¶
The file events to watch for:
- Created
- Newly created files.
- Modified
- Files whose contents are modified. A change to the file's attributes alone does not count.
- Deleted
- When a file is deleted.
- Renamed
- When a file is renamed.
Not every file system can raise every event. When you select a file system, IMan disables the events it cannot report:
| File system | Created | Modified | Deleted | Renamed |
|---|---|---|---|---|
| Default (the IMan server) | Yes | Yes | Yes | Yes |
| AWS S3 | Yes | — | Yes | — |
| Azure Blob Storage | Yes | — | Yes | — |
| OneDrive | Yes | Yes | Yes | — |
The gaps come from the storage services, not from IMan:
- S3 and Azure Blob have no modify or rename. Writing over an existing object replaces it, and the notification that results is a creation. Moving an object is a copy followed by a delete, which reports as those two events.
- OneDrive cannot separate a rename from an edit. The change feed reports a renamed item as an ordinary change, so a rename arrives as Modified.
Watching a cloud file system¶
Change notifications must be enabled¶
A File Event on a cloud file system only fires if that file system has Enable native change notifications ticked on its File Systems record. Notifications are off by default, so you must turn them on.
AWS S3 and Azure Blob Storage also need a queue to carry the events: an SQS queue for S3, or a Storage queue fed by Event Grid for Blob. The bucket or storage account must publish to that queue. You set up both in the provider's console. OneDrive needs no queue.
How often changes are noticed¶
IMan checks for changes on the interval set by the Poll Interval on the file system record, which defaults to 15 seconds. A File Event on a cloud file system therefore triggers within that interval of the change, not immediately.
After a Scheduler restart¶
What happens to changes that occur while the Realisable Scheduler Service is stopped depends on the file system:
-
AWS S3 and Azure Blob Storage lose nothing. The events wait in the SQS or Storage queue until IMan has processed them, and IMan picks up the backlog when the service restarts.
-
OneDrive re-reports the entire drive. IMan does not keep its position in the change feed across a restart. The first poll after a restart lists every file in the drive and reports it. A File Event watching for Created will trigger its integration for all of them.
Build the integration to tolerate a file it has already seen
Because of the OneDrive behaviour above, IMan will from time to time hand an integration triggered from a OneDrive File Event files it has already processed. The integration must cope with that and not assume every file it receives is new.
There are two usual approaches. The integration can move or rename each file out of the watched folder once it has processed it, so a repeat notification finds nothing to do. Or you can key the incoming data so that a repeated file updates the same records instead of creating duplicates.