Problems

• Pre-v10.1.113, IguanaX creates a log_crash_<datetime> folder

• It creates a fresh logt folder

• Queued messages and logs will disappear upon restart of the IguanaX Service since they have moved into the log_crash_<datetime> folder

Causes & Explanation

• Queued messages and logs are stored in the logt folder.

• When pre-v10.1.113 IguanaX starts up, if it crashes during startup, it will rename logt to log_crash_<datetime> and create a new, empty logt folder.

  • If your log directory was in a custom location, it will also move the log_crash_<datetime> into the working directory.

• As a result, pre-crash logs and queued messages are moved to log_crash_<datetime> and IguanaX won't display/process them

Solution

STEP 1: Flush out and process the pre-crash queued messages

• Stop the IguanaX Service

• Rename logt folder to logt_new

• Rename the log_crash_<datetime> folder to logt

  • If your log directory is located outside of your working directory, log_crash_<datetime> will be generated in the working directory. You must rename it to logt and move it into the log directory.

• Start IguanaX in safe mode, see Run Iguana from the command line for running IguanaX in safe mode

Info

The Queue Count on the dashboard may/will likely display the wrong number (often displays 0 even if there is a queue). Do not rely on the Queue Count to be accurate

• To process all queued data from before the crash, turn on all downstream components (components that process queued data). Do not turn on upstream components as these may continue to add to the queue

  • Monitor the logs of all the downstream component(s) to know when the queue is empty. In each component's logs, you will probably see multiple log entries similar to Queue 'Enqueue_5KwNuHVSH3yDHp' is empty. Wait for new messages. Once you see this log message, AND you wait for a few seconds, if nothing else is dequeued after, that specific queue is likely empty

• To process only specific queued data from before the crash, do not turn on components. Go into the logs of the upstream component and manually resubmit any messages you want to get processed by the downstream component(s)

• Stop IguanaX from the command prompt/terminal

• Rename logt back to log_crash_<datetime>

• Rename logt_new back to logt

STEP 2: Merge pre-crash logs with new logs

• Ensure IguanaX is stopped

• Copy and paste over all the component log folders from log_crash_<datetime> to logt (do not copy over the journal file). Be sure you are merging the log folders when you copy/paste, not overriding the folders/files in logt

• Start the IguanaX Service

STEP 3: Restore and Verify

• Verify in your upstream and downstream component logs to see which data has been processed.

Considerations

• We run it in safe mode when flushing out the pre-crash queue to prevent components from auto-starting, so you can have more control over which queues you process.

• If you don't want to bother with flushing out the pre-crash queue, you can skip the “STEP 1: Flush out and process the pre-crash queued messages” and jump to “STEP 2: Merge pre-crash logs with new logs”. Once the IguanaX Service is up and running you would have to manually resubmit messages from before the crash to process them.

• If a component was deleted after the crash but still has queued messages from before the crash, you can process those messages by temporarily restoring the component (works for v 10.1.112):

  • Before starting IguanaX in safe mode (during STEP 1):

    • Copy the deleted component's folder from inside archive > comps

    • Paste it into Working Directory > config > comps

  • Start IguanaX in safe mode and flush the queue as described in STEP 1.

  • Before the final restart in STEP 2:

    • Remove the copied component folder from Working Directory > config > comps

    • This prevents the deleted component from reappearing when IguanaX restarts.

If this solution does not work, contact support for a more manual process to recover old logs/queued messages without stopping IguanaX.