Local File Operations

Lua offers an IO library specific for handling file operations. Our FIL Library is based on these simple functions.

Basic file operations will follow a standard framework:

1. Retrieve Files

Use os.fs.glob() to iterate over files matching a given pattern. In this case, we are iterating over all files with the extension .txt in the defined directory.

-- 1) Retrieve Files   
local sourceDir = '/filePickup/'   
for filePath, fileInfo in os.fs.glob(sourceDir..'*.txt') do   
   -- add file processing here   
end   

2. Process Files

Depending on your workflow you can use the IO library or our FIL Library to:

• Read files - io.read()

• Open files - io.open()

• Write data to files - io.write()

-- 1) Retrieve Files   
local sourceDir = '/filePickup/'   
for filePath, fileInfo in os.fs.glob(sourceDir..'*.txt') do   
   
   -- 2) Open each file to read   
   local F = io.open(filePath, 'r')   
   -- read or write   
   local Content = F:read('*a')   
   F:write('new data')   
   
end   

3. Close File Handles

It is important to close file handles when finished with processing to release system resources and avoid file locks.

-- 1) Retrieve files   
local sourceDir = '/filePickup/'   
for filePath, fileInfo in os.fs.glob(sourceDir..'*.txt') do   
   
   -- 2) Open each file to read   
   local F = io.open(filePath, 'r')   
   -- read or write   
   local Content = F:read('*a')   
   F:write('new data')   
   
   -- 3) Close file handles   
   F:close()   
   
end   

4. Handle Processed Files

Typically, when ingesting files, any processed files should either be moved with os.rename() or deleted from the source directory with os.remove().

-- 1) Retrieve files   
local sourceDir = '/filePickup/'   
for filePath, fileInfo in os.fs.glob(sourceDir..'*.txt') do   
   
   -- 2) Open each file to read   
   local F = io.open(filePath, 'r')   
   -- read or write   
   local Content = F:read('*a')   
   F:write('new data')   
   
   -- 3) Close file handles   
   F:close()   
   
   -- 4) Rename or move a file   
   local filepathTable = filePath:split('/')   
   local filename = filepathTable[#filepathTable]   
   os.rename(filename, '/Users/awiebe/demo/processed/')   
   -- Remove/delete a file   
   os.remove(filePath)   
   
end