mudhorn/n8n
1
0
Fork 0
mirror of https://github.com/n8n-io/n8n.git synced 2024-12-25 04:34:06 -08:00
Code Issues Actions Packages Projects Releases Wiki Activity

fix: Stop OOM crashes in Execution Data pruning (#5095)

Browse source 
* fix: Stop OOM crashed in Execution Data pruning

Currently while pruning execution data, we are loading all the data in memory. For instances where there are thousands of executions, this causes the container to run out of memory.
Since ids is all we need, we should only query for ids.

* query for Executions only when ids are actually needed for pruning binary data

in default mode the binary data is in the database, and will get pruned along with the executions.
This commit is contained in:
कारतोफ्फेलस्क्रिप्ट™ 2023-01-06 11:42:58 +01:00 committed by GitHub
parent f4140d011f
commit c4df2049a8
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 13 additions and 8 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

21
packages/cli/src/WorkflowExecuteAdditionalData.ts
View file

@ -64,7 +64,6 @@ import * as WorkflowHelpers from '@/WorkflowHelpers';
import { getWorkflowOwner } from '@/UserManagement/UserManagementHelper';
import { findSubworkflowStart } from '@/utils';
import { PermissionChecker } from './UserManagement/PermissionChecker';
import { eventBus } from './eventbus';
import { WorkflowsService } from './workflows/workflows.services';
const ERROR_TRIGGER_TYPE = config.getEnv('nodes.errorTriggerType');
@ -204,18 +203,24 @@ async function pruneExecutionData(this: WorkflowHooks): Promise<void> {
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
const utcDate = DateUtils.mixedDateToUtcDatetimeString(date);
const toPrune = { stoppedAt: LessThanOrEqual(utcDate) };
const isBinaryModeDefaultMode = config.getEnv('binaryDataManager.mode') === 'default';
try {
const executions = await Db.collections.Execution.find({
stoppedAt: LessThanOrEqual(utcDate),
});
await Db.collections.Execution.delete({ stoppedAt: LessThanOrEqual(utcDate) });
const executions = isBinaryModeDefaultMode
? []
: await Db.collections.Execution.find({
select: ['id'],
where: toPrune,
});
await Db.collections.Execution.delete(toPrune);
setTimeout(() => {
throttling = false;
}, timeout * 1000);
// Mark binary data for deletion for all executions
await BinaryDataManager.getInstance().markDataForDeletionByExecutionIds(
executions.map(({ id }) => id),
);
if (!isBinaryModeDefaultMode)
await BinaryDataManager.getInstance().markDataForDeletionByExecutionIds(
executions.map(({ id }) => id),
);
} catch (error) {
ErrorReporter.error(error);
throttling = false;