n8n/packages/cli/src/ActiveExecutions.ts

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

217 lines
5.9 KiB
TypeScript
Raw Normal View History

import { Service } from 'typedi';
import type PCancelable from 'p-cancelable';
feat(core): Add execution runData recovery and status field (#5112) * adds ExecutionEvents view modal to ExecutionList * fix time rendering and remove wf column * checks for unfinished executions and fails them * prevent re-setting stoppedAt for execution * some cleanup / manually create rundata after crash * quicksave * remove Threads lib, log worker rewrite * cleanup comment * fix sentry destination return value * test for tests... * run tests with single worker * fix tests * remove console log * add endpoint for execution data recovery * lint cleanup and some refactoring * fix accidental recursion * remove cyclic imports * add rundata recovery to Workflowrunner * remove comments * cleanup and refactor * adds a status field to executions * setExecutionStatus on queued worker * fix onWorkflowPostExecute * set waiting from worker * get crashed status into frontend * remove comment * merge fix * cleanup * catch empty rundata in recovery * refactor IExecutionsSummary and inject nodeExecution Errors * reduce default event log size to 10mb from 100mb * add per node execution status * lint fix * merge and lint fix * phrasing change * improve preview rendering and messaging * remove debug * Improve partial rundata recovery * fix labels * fix line through * send manual rundata to ui at crash * some type and msg push fixes * improve recovered item rendering in preview * update workflowStatistics on recover * merge fix * review fixes * merge fix * notify eventbus when ui is back up * add a small timeout to make sure the UI is back up * increase reconnect timeout to 30s * adjust recover timeout and ui connection lost msg * do not stop execution in editor after x reconnects * add executionRecovered push event * fix recovered connection not green * remove reconnect toast and merge existing rundata * merge editor and recovered data for own mode
2023-02-17 01:54:07 -08:00
import type {
IDeferredPromise,
IExecuteResponsePromiseData,
IRun,
ExecutionStatus,
} from 'n8n-workflow';
import { ApplicationError, createDeferredPromise, sleep } from 'n8n-workflow';
import type {
refactor(core): Implement soft-deletions for executions (#7092) Based on #7065 | Story: https://linear.app/n8n/issue/PAY-771 n8n on filesystem mode marks binary data to delete on manual execution deletion, on unsaved execution completion, and on every execution pruning cycle. We later prune binary data in a separate cycle via these marker files, based on the configured TTL. In the context of introducing an S3 client to manage binary data, the filesystem mode's mark-and-prune setup is too tightly coupled to the general binary data management client interface. This PR... - Ensures the deletion of an execution causes the deletion of any binary data associated to it. This does away with the need for binary data TTL and simplifies the filesystem mode's mark-and-prune setup. - Refactors all execution deletions (including pruning) to cause soft deletions, hard-deletes soft-deleted executions based on the existing pruning config, and adjusts execution endpoints to filter out soft-deleted executions. This reduces DB load, and keeps binary data around long enough for users to access it when building workflows with unsaved executions. - Moves all execution pruning work from an execution lifecycle hook to `execution.repository.ts`. This keeps related logic in a single place. - Removes all marking logic from the binary data manager. This simplifies the interface that the S3 client will meet. - Adds basic sanity-check tests to pruning logic and execution deletion. Out of scope: - Improving existing pruning logic. - Improving existing execution repository logic. - Adjusting dir structure for filesystem mode. --------- Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
2023-09-20 06:21:42 -07:00
ExecutionPayload,
IExecutingWorkflowData,
:sparkles: Unify execution id + Queue system (#1340) * Unify execution ID across executions * Fix indentation and improved comments * WIP: saving data after each node execution * Added on/off to save data after each step, saving initial data and retries working * Fixing lint issues * Fixing more lint issues * :sparkles: Add bull to execute workflows * :shirt: Fix lint issue * :zap: Add graceful shutdown to worker * :zap: Add loading staticData to worker * :shirt: Fix lint issue * :zap: Fix import * Changed tables metadata to add nullable to stoppedAt * Reload database on migration run * Fixed reloading database schema for sqlite by reconnecting and fixing postgres migration * Added checks to Redis and exiting process if connection is unavailable * Fixing error with new installations * Fix issue with data not being sent back to browser on manual executions with defined destination * Merging bull and unify execution id branch fixes * Main process will now get execution success from database instead of redis * Omit execution duration if execution did not stop * Fix issue with execution list displaying inconsistant information information while a workflow is running * Remove unused hooks to clarify for developers that these wont run in queue mode * Added active pooling to help recover from Redis crashes * Lint issues * Changing default polling interval to 60 seconds * Removed unnecessary attributes from bull job * :zap: Improved output on worker job start Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
2021-02-08 23:59:32 -08:00
IExecutionDb,
2020-10-22 06:46:03 -07:00
IExecutionsCurrentSummary,
IWorkflowExecutionDataProcess,
} from '@/Interfaces';
import { isWorkflowIdValid } from '@/utils';
import { ExecutionRepository } from '@db/repositories/execution.repository';
import { Logger } from '@/Logger';
2019-06-23 03:35:23 -07:00
@Service()
2019-06-23 03:35:23 -07:00
export class ActiveExecutions {
private activeExecutions: {
[executionId: string]: IExecutingWorkflowData;
2019-06-23 03:35:23 -07:00
} = {};
constructor(
private readonly logger: Logger,
private readonly executionRepository: ExecutionRepository,
) {}
2019-06-23 03:35:23 -07:00
/**
* Add a new active execution
*/
async add(executionData: IWorkflowExecutionDataProcess, executionId?: string): Promise<string> {
feat(core): Add execution runData recovery and status field (#5112) * adds ExecutionEvents view modal to ExecutionList * fix time rendering and remove wf column * checks for unfinished executions and fails them * prevent re-setting stoppedAt for execution * some cleanup / manually create rundata after crash * quicksave * remove Threads lib, log worker rewrite * cleanup comment * fix sentry destination return value * test for tests... * run tests with single worker * fix tests * remove console log * add endpoint for execution data recovery * lint cleanup and some refactoring * fix accidental recursion * remove cyclic imports * add rundata recovery to Workflowrunner * remove comments * cleanup and refactor * adds a status field to executions * setExecutionStatus on queued worker * fix onWorkflowPostExecute * set waiting from worker * get crashed status into frontend * remove comment * merge fix * cleanup * catch empty rundata in recovery * refactor IExecutionsSummary and inject nodeExecution Errors * reduce default event log size to 10mb from 100mb * add per node execution status * lint fix * merge and lint fix * phrasing change * improve preview rendering and messaging * remove debug * Improve partial rundata recovery * fix labels * fix line through * send manual rundata to ui at crash * some type and msg push fixes * improve recovered item rendering in preview * update workflowStatistics on recover * merge fix * review fixes * merge fix * notify eventbus when ui is back up * add a small timeout to make sure the UI is back up * increase reconnect timeout to 30s * adjust recover timeout and ui connection lost msg * do not stop execution in editor after x reconnects * add executionRecovered push event * fix recovered connection not green * remove reconnect toast and merge existing rundata * merge editor and recovered data for own mode
2023-02-17 01:54:07 -08:00
let executionStatus: ExecutionStatus = executionId ? 'running' : 'new';
:sparkles: Implement Wait functionality (#1817) * refactor saving * refactor api layer to be stateless * refactor header details * set variable for menu height * clean up scss * clean up indentation * clean up dropdown impl * refactor no tags view * split away header * Fix tslint issues * Refactor tag manager * add tags to patch request * clean up scss * :zap: Refactor types to entities * fix issues * update no workflow error * clean up tagscontainer * use getters instead of state * remove imports * use custom colors * clean up tags container * clean up dropdown * clean up focusoncreate * :zap: Ignore mistaken ID in POST /workflows * :zap: Fix undefined tag ID in PATCH /workflows * :zap: Shorten response for POST /tags * remove scss mixins * clean up imports * :zap: Implement validation with class-validator * address ivan's comments * implement modals * Fix lint issues * fix disabling shortcuts * fix focus issues * fix focus issues * fix focus issues with modal * fix linting issues * use dispatch * use constants for modal keys * fix focus * fix lint issues * remove unused prop * add modal root * fix lint issues * remove unused methods * fix shortcut * remove max width * :zap: Fix duplicate entry error for pg and MySQL * update rename messaging * update order of buttons * fix firefox overflow on windows * fix dropdown height * :hammer: refactor tag crud controllers * 🧹 remove unused imports * use variable for number of items * fix dropdown spacing * :zap: Restore type to fix build * :zap: Fix post-refactor PATCH /workflows/:id * :zap: Fix PATCH /workflows/:id for zero tags * :zap: Fix usage count becoming stringified * address max's comments * fix filter spacing * fix blur bug * address most of ivan's comments * address tags type concern * remove defaults * :zap: return tag id as string * :hammer: add hooks to tag CUD operations * 🏎 simplify timestamp pruning * remove blur event * fix onblur bug * :zap: Fix fs import to fix build * address max's comments * implement responsive tag container * fix lint issues * update tag limits * address ivan's comments * remove rename, refactor header, implement new designs for save, remove responsive tag container * update styling * update styling * implement responsive tag container * implement header tags edit * implement header tags edit * fix lint issues * implement expandable input * minor fixes * minor fixes * use variable * rename save as * duplicate fixes * minor edit fixes * lint fixes * style fixes * hook up saving name * hook up tags * clean up impl * fix dirty state bug * update limit * update notification messages * on click outside * fix minor bug with count * lint fixes * handle minor edge cases * handle minor edge cases * handle minor bugs; fix firefox dropdown issue * Fix min width * apply tags only after api success * remove count fix * clean up workflow tags impl, fix tags delete bug * fix minor issue * fix minor spacing issue * disable wrap for ops * fix viewport root; save on click in dropdown * save button loading when saving name/tags * implement max width on tags container * implement cleaner create experience * disable edit while updating * codacy hex color * refactor tags container * fix clickability * fix workflow open and count * clean up structure * fix up lint issues * fix button size * increase workflow name limit for larger screen * tslint fixes * disable responsiveness for workflow modal * rename event * change min width for tags * clean up pr * address max's comments on styles * remove success toasts * add hover mode to name * minor fixes * refactor name preview * fix name input not to jiggle * finish up name input * Fix up add tags * clean up param * clean up scss * fix resizing name * fix resizing name * fix resize bug * clean up edit spacing * ignore on esc * fix input bug * focus input on clear * build * fix up add tags clickablity * remove scrollbars * move into folders * clean up multiple patch req * remove padding top from edit * update tags on enter * build * rollout blur on enter behavior * rollout esc behavior * fix tags bug when duplicating tags * move key to reload tags * update header spacing * build * update hex case * refactor workflow title * remove unusued prop * keep focus on error, fix bug on error * Fix bug with name / tags toggle on error * fix connection push bug * :spakles: Implement wait functionality * :bug: Do not delete waiting executions with prune * :zap: Improve SQLite migration to not lose execution data anymore * :zap: Make it possible to restart waiting execution via webhook * :zap: Add missing file * :bug: Some more merge fixes * :zap: Do not show error for Wait-Nodes if in time-mode * :zap: Make $executionId available in expressions * :shirt: Fix lint issue * :shirt: Fix lint issue * :shirt: Fix lint issue * :zap: Set the unlimited sleep time as a variable * :zap: Add also sleeping webhook path to config * :zap: Make it possible to retrieve restartUrl in workflow * :zap: Add authentication to Wait-Node in Webhook-Mode * :zap: Return 404 when trying to restart execution via webhook which does not support it * :sparkles: Make it possible to set absolute time on Wait-Node * :zap: Remove not needed imports * :zap: Fix description format * :sparkles: Implement missing webhook features on Wait-Node * :zap: Display webhook variable in NodeWebhooks * :zap: Include also date in displayed sleep time * :zap: Make it possible to see sleep time on node * :zap: Make sure that no executions does get executed twice * :zap: Add comment * :zap: Further improvements * :zap: Make Wait-Node easier to use * :sparkles: Add support for "notice" parameter type * Fixing wait node to work with queue, improved logging and execution view * Added support for mysql and pg * :sparkles: Add support for webhook postfix path * :sparkles: Make it possible to stop sleeping executions * :zap: Fix issue with webhook paths in not webhook mode * :zap: Remove not needed console.log * :zap: Update TODOs * :zap: Increase min time of workflow staying active to descrease possible issue with overlap * :shirt: Fix lint issue * :bug: Fix issues with webhooks * :zap: Make error message clearer * :zap: Fix issue with missing execution ID in scaling mode * Fixed execution list to correctly display waiting executins * Feature: enable webhook wait workflows to continue after specified time * Fixed linting * :zap: Improve waiting description text * :zap: Fix parameter display issue and rename * :zap: Remove comment * :zap: Do not display webhooks on Wait-Node * Changed wording from restart to resume on wait node * Fixed wording and inconsistent screen when changing resume modes * Removed dots from the descriptions * Changed docs url and renaming postfix to suffix * Changed names from sleep to wait * :zap: Apply suggestions from ben Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> * Some fixes by Ben * :zap: Remove console.logs * :zap: Fixes and improvements Co-authored-by: Mutasem <mutdmour@gmail.com> Co-authored-by: Iván Ovejero <ivov.src@gmail.com> Co-authored-by: Ben Hesseldieck <b.hesseldieck@gmail.com> Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> Co-authored-by: Omar Ajoue <krynble@gmail.com>
2021-08-21 05:11:32 -07:00
if (executionId === undefined) {
// Is a new execution so save in DB
:sparkles: Unify execution id + Queue system (#1340) * Unify execution ID across executions * Fix indentation and improved comments * WIP: saving data after each node execution * Added on/off to save data after each step, saving initial data and retries working * Fixing lint issues * Fixing more lint issues * :sparkles: Add bull to execute workflows * :shirt: Fix lint issue * :zap: Add graceful shutdown to worker * :zap: Add loading staticData to worker * :shirt: Fix lint issue * :zap: Fix import * Changed tables metadata to add nullable to stoppedAt * Reload database on migration run * Fixed reloading database schema for sqlite by reconnecting and fixing postgres migration * Added checks to Redis and exiting process if connection is unavailable * Fixing error with new installations * Fix issue with data not being sent back to browser on manual executions with defined destination * Merging bull and unify execution id branch fixes * Main process will now get execution success from database instead of redis * Omit execution duration if execution did not stop * Fix issue with execution list displaying inconsistant information information while a workflow is running * Remove unused hooks to clarify for developers that these wont run in queue mode * Added active pooling to help recover from Redis crashes * Lint issues * Changing default polling interval to 60 seconds * Removed unnecessary attributes from bull job * :zap: Improved output on worker job start Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
2021-02-08 23:59:32 -08:00
refactor(core): Implement soft-deletions for executions (#7092) Based on #7065 | Story: https://linear.app/n8n/issue/PAY-771 n8n on filesystem mode marks binary data to delete on manual execution deletion, on unsaved execution completion, and on every execution pruning cycle. We later prune binary data in a separate cycle via these marker files, based on the configured TTL. In the context of introducing an S3 client to manage binary data, the filesystem mode's mark-and-prune setup is too tightly coupled to the general binary data management client interface. This PR... - Ensures the deletion of an execution causes the deletion of any binary data associated to it. This does away with the need for binary data TTL and simplifies the filesystem mode's mark-and-prune setup. - Refactors all execution deletions (including pruning) to cause soft deletions, hard-deletes soft-deleted executions based on the existing pruning config, and adjusts execution endpoints to filter out soft-deleted executions. This reduces DB load, and keeps binary data around long enough for users to access it when building workflows with unsaved executions. - Moves all execution pruning work from an execution lifecycle hook to `execution.repository.ts`. This keeps related logic in a single place. - Removes all marking logic from the binary data manager. This simplifies the interface that the S3 client will meet. - Adds basic sanity-check tests to pruning logic and execution deletion. Out of scope: - Improving existing pruning logic. - Improving existing execution repository logic. - Adjusting dir structure for filesystem mode. --------- Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
2023-09-20 06:21:42 -07:00
const fullExecutionData: ExecutionPayload = {
:sparkles: Implement Wait functionality (#1817) * refactor saving * refactor api layer to be stateless * refactor header details * set variable for menu height * clean up scss * clean up indentation * clean up dropdown impl * refactor no tags view * split away header * Fix tslint issues * Refactor tag manager * add tags to patch request * clean up scss * :zap: Refactor types to entities * fix issues * update no workflow error * clean up tagscontainer * use getters instead of state * remove imports * use custom colors * clean up tags container * clean up dropdown * clean up focusoncreate * :zap: Ignore mistaken ID in POST /workflows * :zap: Fix undefined tag ID in PATCH /workflows * :zap: Shorten response for POST /tags * remove scss mixins * clean up imports * :zap: Implement validation with class-validator * address ivan's comments * implement modals * Fix lint issues * fix disabling shortcuts * fix focus issues * fix focus issues * fix focus issues with modal * fix linting issues * use dispatch * use constants for modal keys * fix focus * fix lint issues * remove unused prop * add modal root * fix lint issues * remove unused methods * fix shortcut * remove max width * :zap: Fix duplicate entry error for pg and MySQL * update rename messaging * update order of buttons * fix firefox overflow on windows * fix dropdown height * :hammer: refactor tag crud controllers * 🧹 remove unused imports * use variable for number of items * fix dropdown spacing * :zap: Restore type to fix build * :zap: Fix post-refactor PATCH /workflows/:id * :zap: Fix PATCH /workflows/:id for zero tags * :zap: Fix usage count becoming stringified * address max's comments * fix filter spacing * fix blur bug * address most of ivan's comments * address tags type concern * remove defaults * :zap: return tag id as string * :hammer: add hooks to tag CUD operations * 🏎 simplify timestamp pruning * remove blur event * fix onblur bug * :zap: Fix fs import to fix build * address max's comments * implement responsive tag container * fix lint issues * update tag limits * address ivan's comments * remove rename, refactor header, implement new designs for save, remove responsive tag container * update styling * update styling * implement responsive tag container * implement header tags edit * implement header tags edit * fix lint issues * implement expandable input * minor fixes * minor fixes * use variable * rename save as * duplicate fixes * minor edit fixes * lint fixes * style fixes * hook up saving name * hook up tags * clean up impl * fix dirty state bug * update limit * update notification messages * on click outside * fix minor bug with count * lint fixes * handle minor edge cases * handle minor edge cases * handle minor bugs; fix firefox dropdown issue * Fix min width * apply tags only after api success * remove count fix * clean up workflow tags impl, fix tags delete bug * fix minor issue * fix minor spacing issue * disable wrap for ops * fix viewport root; save on click in dropdown * save button loading when saving name/tags * implement max width on tags container * implement cleaner create experience * disable edit while updating * codacy hex color * refactor tags container * fix clickability * fix workflow open and count * clean up structure * fix up lint issues * fix button size * increase workflow name limit for larger screen * tslint fixes * disable responsiveness for workflow modal * rename event * change min width for tags * clean up pr * address max's comments on styles * remove success toasts * add hover mode to name * minor fixes * refactor name preview * fix name input not to jiggle * finish up name input * Fix up add tags * clean up param * clean up scss * fix resizing name * fix resizing name * fix resize bug * clean up edit spacing * ignore on esc * fix input bug * focus input on clear * build * fix up add tags clickablity * remove scrollbars * move into folders * clean up multiple patch req * remove padding top from edit * update tags on enter * build * rollout blur on enter behavior * rollout esc behavior * fix tags bug when duplicating tags * move key to reload tags * update header spacing * build * update hex case * refactor workflow title * remove unusued prop * keep focus on error, fix bug on error * Fix bug with name / tags toggle on error * fix connection push bug * :spakles: Implement wait functionality * :bug: Do not delete waiting executions with prune * :zap: Improve SQLite migration to not lose execution data anymore * :zap: Make it possible to restart waiting execution via webhook * :zap: Add missing file * :bug: Some more merge fixes * :zap: Do not show error for Wait-Nodes if in time-mode * :zap: Make $executionId available in expressions * :shirt: Fix lint issue * :shirt: Fix lint issue * :shirt: Fix lint issue * :zap: Set the unlimited sleep time as a variable * :zap: Add also sleeping webhook path to config * :zap: Make it possible to retrieve restartUrl in workflow * :zap: Add authentication to Wait-Node in Webhook-Mode * :zap: Return 404 when trying to restart execution via webhook which does not support it * :sparkles: Make it possible to set absolute time on Wait-Node * :zap: Remove not needed imports * :zap: Fix description format * :sparkles: Implement missing webhook features on Wait-Node * :zap: Display webhook variable in NodeWebhooks * :zap: Include also date in displayed sleep time * :zap: Make it possible to see sleep time on node * :zap: Make sure that no executions does get executed twice * :zap: Add comment * :zap: Further improvements * :zap: Make Wait-Node easier to use * :sparkles: Add support for "notice" parameter type * Fixing wait node to work with queue, improved logging and execution view * Added support for mysql and pg * :sparkles: Add support for webhook postfix path * :sparkles: Make it possible to stop sleeping executions * :zap: Fix issue with webhook paths in not webhook mode * :zap: Remove not needed console.log * :zap: Update TODOs * :zap: Increase min time of workflow staying active to descrease possible issue with overlap * :shirt: Fix lint issue * :bug: Fix issues with webhooks * :zap: Make error message clearer * :zap: Fix issue with missing execution ID in scaling mode * Fixed execution list to correctly display waiting executins * Feature: enable webhook wait workflows to continue after specified time * Fixed linting * :zap: Improve waiting description text * :zap: Fix parameter display issue and rename * :zap: Remove comment * :zap: Do not display webhooks on Wait-Node * Changed wording from restart to resume on wait node * Fixed wording and inconsistent screen when changing resume modes * Removed dots from the descriptions * Changed docs url and renaming postfix to suffix * Changed names from sleep to wait * :zap: Apply suggestions from ben Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> * Some fixes by Ben * :zap: Remove console.logs * :zap: Fixes and improvements Co-authored-by: Mutasem <mutdmour@gmail.com> Co-authored-by: Iván Ovejero <ivov.src@gmail.com> Co-authored-by: Ben Hesseldieck <b.hesseldieck@gmail.com> Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> Co-authored-by: Omar Ajoue <krynble@gmail.com>
2021-08-21 05:11:32 -07:00
data: executionData.executionData!,
mode: executionData.executionMode,
finished: false,
startedAt: new Date(),
workflowData: executionData.workflowData,
feat(core): Add execution runData recovery and status field (#5112) * adds ExecutionEvents view modal to ExecutionList * fix time rendering and remove wf column * checks for unfinished executions and fails them * prevent re-setting stoppedAt for execution * some cleanup / manually create rundata after crash * quicksave * remove Threads lib, log worker rewrite * cleanup comment * fix sentry destination return value * test for tests... * run tests with single worker * fix tests * remove console log * add endpoint for execution data recovery * lint cleanup and some refactoring * fix accidental recursion * remove cyclic imports * add rundata recovery to Workflowrunner * remove comments * cleanup and refactor * adds a status field to executions * setExecutionStatus on queued worker * fix onWorkflowPostExecute * set waiting from worker * get crashed status into frontend * remove comment * merge fix * cleanup * catch empty rundata in recovery * refactor IExecutionsSummary and inject nodeExecution Errors * reduce default event log size to 10mb from 100mb * add per node execution status * lint fix * merge and lint fix * phrasing change * improve preview rendering and messaging * remove debug * Improve partial rundata recovery * fix labels * fix line through * send manual rundata to ui at crash * some type and msg push fixes * improve recovered item rendering in preview * update workflowStatistics on recover * merge fix * review fixes * merge fix * notify eventbus when ui is back up * add a small timeout to make sure the UI is back up * increase reconnect timeout to 30s * adjust recover timeout and ui connection lost msg * do not stop execution in editor after x reconnects * add executionRecovered push event * fix recovered connection not green * remove reconnect toast and merge existing rundata * merge editor and recovered data for own mode
2023-02-17 01:54:07 -08:00
status: executionStatus,
workflowId: executionData.workflowData.id,
:sparkles: Implement Wait functionality (#1817) * refactor saving * refactor api layer to be stateless * refactor header details * set variable for menu height * clean up scss * clean up indentation * clean up dropdown impl * refactor no tags view * split away header * Fix tslint issues * Refactor tag manager * add tags to patch request * clean up scss * :zap: Refactor types to entities * fix issues * update no workflow error * clean up tagscontainer * use getters instead of state * remove imports * use custom colors * clean up tags container * clean up dropdown * clean up focusoncreate * :zap: Ignore mistaken ID in POST /workflows * :zap: Fix undefined tag ID in PATCH /workflows * :zap: Shorten response for POST /tags * remove scss mixins * clean up imports * :zap: Implement validation with class-validator * address ivan's comments * implement modals * Fix lint issues * fix disabling shortcuts * fix focus issues * fix focus issues * fix focus issues with modal * fix linting issues * use dispatch * use constants for modal keys * fix focus * fix lint issues * remove unused prop * add modal root * fix lint issues * remove unused methods * fix shortcut * remove max width * :zap: Fix duplicate entry error for pg and MySQL * update rename messaging * update order of buttons * fix firefox overflow on windows * fix dropdown height * :hammer: refactor tag crud controllers * 🧹 remove unused imports * use variable for number of items * fix dropdown spacing * :zap: Restore type to fix build * :zap: Fix post-refactor PATCH /workflows/:id * :zap: Fix PATCH /workflows/:id for zero tags * :zap: Fix usage count becoming stringified * address max's comments * fix filter spacing * fix blur bug * address most of ivan's comments * address tags type concern * remove defaults * :zap: return tag id as string * :hammer: add hooks to tag CUD operations * 🏎 simplify timestamp pruning * remove blur event * fix onblur bug * :zap: Fix fs import to fix build * address max's comments * implement responsive tag container * fix lint issues * update tag limits * address ivan's comments * remove rename, refactor header, implement new designs for save, remove responsive tag container * update styling * update styling * implement responsive tag container * implement header tags edit * implement header tags edit * fix lint issues * implement expandable input * minor fixes * minor fixes * use variable * rename save as * duplicate fixes * minor edit fixes * lint fixes * style fixes * hook up saving name * hook up tags * clean up impl * fix dirty state bug * update limit * update notification messages * on click outside * fix minor bug with count * lint fixes * handle minor edge cases * handle minor edge cases * handle minor bugs; fix firefox dropdown issue * Fix min width * apply tags only after api success * remove count fix * clean up workflow tags impl, fix tags delete bug * fix minor issue * fix minor spacing issue * disable wrap for ops * fix viewport root; save on click in dropdown * save button loading when saving name/tags * implement max width on tags container * implement cleaner create experience * disable edit while updating * codacy hex color * refactor tags container * fix clickability * fix workflow open and count * clean up structure * fix up lint issues * fix button size * increase workflow name limit for larger screen * tslint fixes * disable responsiveness for workflow modal * rename event * change min width for tags * clean up pr * address max's comments on styles * remove success toasts * add hover mode to name * minor fixes * refactor name preview * fix name input not to jiggle * finish up name input * Fix up add tags * clean up param * clean up scss * fix resizing name * fix resizing name * fix resize bug * clean up edit spacing * ignore on esc * fix input bug * focus input on clear * build * fix up add tags clickablity * remove scrollbars * move into folders * clean up multiple patch req * remove padding top from edit * update tags on enter * build * rollout blur on enter behavior * rollout esc behavior * fix tags bug when duplicating tags * move key to reload tags * update header spacing * build * update hex case * refactor workflow title * remove unusued prop * keep focus on error, fix bug on error * Fix bug with name / tags toggle on error * fix connection push bug * :spakles: Implement wait functionality * :bug: Do not delete waiting executions with prune * :zap: Improve SQLite migration to not lose execution data anymore * :zap: Make it possible to restart waiting execution via webhook * :zap: Add missing file * :bug: Some more merge fixes * :zap: Do not show error for Wait-Nodes if in time-mode * :zap: Make $executionId available in expressions * :shirt: Fix lint issue * :shirt: Fix lint issue * :shirt: Fix lint issue * :zap: Set the unlimited sleep time as a variable * :zap: Add also sleeping webhook path to config * :zap: Make it possible to retrieve restartUrl in workflow * :zap: Add authentication to Wait-Node in Webhook-Mode * :zap: Return 404 when trying to restart execution via webhook which does not support it * :sparkles: Make it possible to set absolute time on Wait-Node * :zap: Remove not needed imports * :zap: Fix description format * :sparkles: Implement missing webhook features on Wait-Node * :zap: Display webhook variable in NodeWebhooks * :zap: Include also date in displayed sleep time * :zap: Make it possible to see sleep time on node * :zap: Make sure that no executions does get executed twice * :zap: Add comment * :zap: Further improvements * :zap: Make Wait-Node easier to use * :sparkles: Add support for "notice" parameter type * Fixing wait node to work with queue, improved logging and execution view * Added support for mysql and pg * :sparkles: Add support for webhook postfix path * :sparkles: Make it possible to stop sleeping executions * :zap: Fix issue with webhook paths in not webhook mode * :zap: Remove not needed console.log * :zap: Update TODOs * :zap: Increase min time of workflow staying active to descrease possible issue with overlap * :shirt: Fix lint issue * :bug: Fix issues with webhooks * :zap: Make error message clearer * :zap: Fix issue with missing execution ID in scaling mode * Fixed execution list to correctly display waiting executins * Feature: enable webhook wait workflows to continue after specified time * Fixed linting * :zap: Improve waiting description text * :zap: Fix parameter display issue and rename * :zap: Remove comment * :zap: Do not display webhooks on Wait-Node * Changed wording from restart to resume on wait node * Fixed wording and inconsistent screen when changing resume modes * Removed dots from the descriptions * Changed docs url and renaming postfix to suffix * Changed names from sleep to wait * :zap: Apply suggestions from ben Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> * Some fixes by Ben * :zap: Remove console.logs * :zap: Fixes and improvements Co-authored-by: Mutasem <mutdmour@gmail.com> Co-authored-by: Iván Ovejero <ivov.src@gmail.com> Co-authored-by: Ben Hesseldieck <b.hesseldieck@gmail.com> Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> Co-authored-by: Omar Ajoue <krynble@gmail.com>
2021-08-21 05:11:32 -07:00
};
:sparkles: Unify execution id + Queue system (#1340) * Unify execution ID across executions * Fix indentation and improved comments * WIP: saving data after each node execution * Added on/off to save data after each step, saving initial data and retries working * Fixing lint issues * Fixing more lint issues * :sparkles: Add bull to execute workflows * :shirt: Fix lint issue * :zap: Add graceful shutdown to worker * :zap: Add loading staticData to worker * :shirt: Fix lint issue * :zap: Fix import * Changed tables metadata to add nullable to stoppedAt * Reload database on migration run * Fixed reloading database schema for sqlite by reconnecting and fixing postgres migration * Added checks to Redis and exiting process if connection is unavailable * Fixing error with new installations * Fix issue with data not being sent back to browser on manual executions with defined destination * Merging bull and unify execution id branch fixes * Main process will now get execution success from database instead of redis * Omit execution duration if execution did not stop * Fix issue with execution list displaying inconsistant information information while a workflow is running * Remove unused hooks to clarify for developers that these wont run in queue mode * Added active pooling to help recover from Redis crashes * Lint issues * Changing default polling interval to 60 seconds * Removed unnecessary attributes from bull job * :zap: Improved output on worker job start Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
2021-02-08 23:59:32 -08:00
:sparkles: Implement Wait functionality (#1817) * refactor saving * refactor api layer to be stateless * refactor header details * set variable for menu height * clean up scss * clean up indentation * clean up dropdown impl * refactor no tags view * split away header * Fix tslint issues * Refactor tag manager * add tags to patch request * clean up scss * :zap: Refactor types to entities * fix issues * update no workflow error * clean up tagscontainer * use getters instead of state * remove imports * use custom colors * clean up tags container * clean up dropdown * clean up focusoncreate * :zap: Ignore mistaken ID in POST /workflows * :zap: Fix undefined tag ID in PATCH /workflows * :zap: Shorten response for POST /tags * remove scss mixins * clean up imports * :zap: Implement validation with class-validator * address ivan's comments * implement modals * Fix lint issues * fix disabling shortcuts * fix focus issues * fix focus issues * fix focus issues with modal * fix linting issues * use dispatch * use constants for modal keys * fix focus * fix lint issues * remove unused prop * add modal root * fix lint issues * remove unused methods * fix shortcut * remove max width * :zap: Fix duplicate entry error for pg and MySQL * update rename messaging * update order of buttons * fix firefox overflow on windows * fix dropdown height * :hammer: refactor tag crud controllers * 🧹 remove unused imports * use variable for number of items * fix dropdown spacing * :zap: Restore type to fix build * :zap: Fix post-refactor PATCH /workflows/:id * :zap: Fix PATCH /workflows/:id for zero tags * :zap: Fix usage count becoming stringified * address max's comments * fix filter spacing * fix blur bug * address most of ivan's comments * address tags type concern * remove defaults * :zap: return tag id as string * :hammer: add hooks to tag CUD operations * 🏎 simplify timestamp pruning * remove blur event * fix onblur bug * :zap: Fix fs import to fix build * address max's comments * implement responsive tag container * fix lint issues * update tag limits * address ivan's comments * remove rename, refactor header, implement new designs for save, remove responsive tag container * update styling * update styling * implement responsive tag container * implement header tags edit * implement header tags edit * fix lint issues * implement expandable input * minor fixes * minor fixes * use variable * rename save as * duplicate fixes * minor edit fixes * lint fixes * style fixes * hook up saving name * hook up tags * clean up impl * fix dirty state bug * update limit * update notification messages * on click outside * fix minor bug with count * lint fixes * handle minor edge cases * handle minor edge cases * handle minor bugs; fix firefox dropdown issue * Fix min width * apply tags only after api success * remove count fix * clean up workflow tags impl, fix tags delete bug * fix minor issue * fix minor spacing issue * disable wrap for ops * fix viewport root; save on click in dropdown * save button loading when saving name/tags * implement max width on tags container * implement cleaner create experience * disable edit while updating * codacy hex color * refactor tags container * fix clickability * fix workflow open and count * clean up structure * fix up lint issues * fix button size * increase workflow name limit for larger screen * tslint fixes * disable responsiveness for workflow modal * rename event * change min width for tags * clean up pr * address max's comments on styles * remove success toasts * add hover mode to name * minor fixes * refactor name preview * fix name input not to jiggle * finish up name input * Fix up add tags * clean up param * clean up scss * fix resizing name * fix resizing name * fix resize bug * clean up edit spacing * ignore on esc * fix input bug * focus input on clear * build * fix up add tags clickablity * remove scrollbars * move into folders * clean up multiple patch req * remove padding top from edit * update tags on enter * build * rollout blur on enter behavior * rollout esc behavior * fix tags bug when duplicating tags * move key to reload tags * update header spacing * build * update hex case * refactor workflow title * remove unusued prop * keep focus on error, fix bug on error * Fix bug with name / tags toggle on error * fix connection push bug * :spakles: Implement wait functionality * :bug: Do not delete waiting executions with prune * :zap: Improve SQLite migration to not lose execution data anymore * :zap: Make it possible to restart waiting execution via webhook * :zap: Add missing file * :bug: Some more merge fixes * :zap: Do not show error for Wait-Nodes if in time-mode * :zap: Make $executionId available in expressions * :shirt: Fix lint issue * :shirt: Fix lint issue * :shirt: Fix lint issue * :zap: Set the unlimited sleep time as a variable * :zap: Add also sleeping webhook path to config * :zap: Make it possible to retrieve restartUrl in workflow * :zap: Add authentication to Wait-Node in Webhook-Mode * :zap: Return 404 when trying to restart execution via webhook which does not support it * :sparkles: Make it possible to set absolute time on Wait-Node * :zap: Remove not needed imports * :zap: Fix description format * :sparkles: Implement missing webhook features on Wait-Node * :zap: Display webhook variable in NodeWebhooks * :zap: Include also date in displayed sleep time * :zap: Make it possible to see sleep time on node * :zap: Make sure that no executions does get executed twice * :zap: Add comment * :zap: Further improvements * :zap: Make Wait-Node easier to use * :sparkles: Add support for "notice" parameter type * Fixing wait node to work with queue, improved logging and execution view * Added support for mysql and pg * :sparkles: Add support for webhook postfix path * :sparkles: Make it possible to stop sleeping executions * :zap: Fix issue with webhook paths in not webhook mode * :zap: Remove not needed console.log * :zap: Update TODOs * :zap: Increase min time of workflow staying active to descrease possible issue with overlap * :shirt: Fix lint issue * :bug: Fix issues with webhooks * :zap: Make error message clearer * :zap: Fix issue with missing execution ID in scaling mode * Fixed execution list to correctly display waiting executins * Feature: enable webhook wait workflows to continue after specified time * Fixed linting * :zap: Improve waiting description text * :zap: Fix parameter display issue and rename * :zap: Remove comment * :zap: Do not display webhooks on Wait-Node * Changed wording from restart to resume on wait node * Fixed wording and inconsistent screen when changing resume modes * Removed dots from the descriptions * Changed docs url and renaming postfix to suffix * Changed names from sleep to wait * :zap: Apply suggestions from ben Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> * Some fixes by Ben * :zap: Remove console.logs * :zap: Fixes and improvements Co-authored-by: Mutasem <mutdmour@gmail.com> Co-authored-by: Iván Ovejero <ivov.src@gmail.com> Co-authored-by: Ben Hesseldieck <b.hesseldieck@gmail.com> Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> Co-authored-by: Omar Ajoue <krynble@gmail.com>
2021-08-21 05:11:32 -07:00
if (executionData.retryOf !== undefined) {
fullExecutionData.retryOf = executionData.retryOf.toString();
}
:sparkles: Add tagging of workflows (#1647) * clean up dropdown * clean up focusoncreate * :zap: Ignore mistaken ID in POST /workflows * :zap: Fix undefined tag ID in PATCH /workflows * :zap: Shorten response for POST /tags * remove scss mixins * clean up imports * :zap: Implement validation with class-validator * address ivan's comments * implement modals * Fix lint issues * fix disabling shortcuts * fix focus issues * fix focus issues * fix focus issues with modal * fix linting issues * use dispatch * use constants for modal keys * fix focus * fix lint issues * remove unused prop * add modal root * fix lint issues * remove unused methods * fix shortcut * remove max width * :zap: Fix duplicate entry error for pg and MySQL * update rename messaging * update order of buttons * fix firefox overflow on windows * fix dropdown height * :hammer: refactor tag crud controllers * 🧹 remove unused imports * use variable for number of items * fix dropdown spacing * :zap: Restore type to fix build * :zap: Fix post-refactor PATCH /workflows/:id * :zap: Fix PATCH /workflows/:id for zero tags * :zap: Fix usage count becoming stringified * address max's comments * fix filter spacing * fix blur bug * address most of ivan's comments * address tags type concern * remove defaults * :zap: return tag id as string * :hammer: add hooks to tag CUD operations * 🏎 simplify timestamp pruning * remove blur event * fix onblur bug * :zap: Fix fs import to fix build * address max's comments * implement responsive tag container * fix lint issues * Set default dates in entities * :shirt: Fix lint in migrations * update tag limits * address ivan's comments * remove rename, refactor header, implement new designs for save, remove responsive tag container * update styling * update styling * implement responsive tag container * implement header tags edit * implement header tags edit * fix lint issues * implement expandable input * minor fixes * minor fixes * use variable * rename save as * duplicate fixes * :zap: Implement unique workflow names * :zap: Create /workflows/new endpoint * minor edit fixes * lint fixes * style fixes * hook up saving name * hook up tags * clean up impl * fix dirty state bug * update limit * update notification messages * on click outside * fix minor bug with count * lint fixes * :zap: Add query string params to /workflows/new * handle minor edge cases * handle minor edge cases * handle minor bugs; fix firefox dropdown issue * Fix min width * apply tags only after api success * remove count fix * :construction: Adjust to new qs requirements * clean up workflow tags impl, fix tags delete bug * fix minor issue * fix minor spacing issue * disable wrap for ops * fix viewport root; save on click in dropdown * save button loading when saving name/tags * implement max width on tags container * implement cleaner create experience * disable edit while updating * codacy hex color * refactor tags container * fix clickability * fix workflow open and count * clean up structure * fix up lint issues * :zap: Create migrations for unique workflow names * fix button size * increase workflow name limit for larger screen * tslint fixes * disable responsiveness for workflow modal * rename event * change min width for tags * clean up pr * :zap: Adjust quotes in MySQL migration * :zap: Adjust quotes in Postgres migration * address max's comments on styles * remove success toasts * add hover mode to name * minor fixes * refactor name preview * fix name input not to jiggle * finish up name input * Fix up add tags * clean up param * clean up scss * fix resizing name * fix resizing name * fix resize bug * clean up edit spacing * ignore on esc * fix input bug * focus input on clear * build * fix up add tags clickablity * remove scrollbars * move into folders * clean up multiple patch req * remove padding top from edit * update tags on enter * build * rollout blur on enter behavior * rollout esc behavior * fix tags bug when duplicating tags * move key to reload tags * update header spacing * build * update hex case * refactor workflow title * remove unusued prop * keep focus on error, fix bug on error * Fix bug with name / tags toggle on error * impl creating new workflow name * :zap: Refactor endpoint per new guidelines * support naming endpoint * :zap: Refactor to support numeric suffixes * :shirt: Lint migrations for unique workflow names * :zap: Add migrations set default dates to indexes * fix connection push bug * :zap: Lowercase default workflow name * :zap: Add prefixes to set default dates migration * :zap: Fix indentation on default dates migrations * :zap: Add temp ts-ignore for unrelated change * :zap: Adjust default dates migration for MySQL Remove change to data column in credentials_entity, already covered by Omar's migration. Also, fix quotes from table prefix addition. * :zap: Adjust quotes in dates migration for PG * fix safari color bug * fix count bug * fix scroll bugs in dropdown * expand filter size * apply box-sizing to main header * update workflow names in executions to be wrapped by quotes * fix bug where key is same in dropdown * fix firefox bug * move up push connection session * :hammer: Remove mistakenly added nullable property * :fire: Remove unneeded index drop-create (PG) * :fire: Remove unneeded table copying * :zap: Merge dates migration with tags migration * :hammer: Refactor endpoint and make wf name env * dropdown colors in firefox * update colors to use variables * update thumb color * change error message * remove 100 char maximum * fix bug with saving tags dropdowns multiple times * update error message when no name * :zap: Update name missing toast message * :zap: Update workflow already exists message * disable saving for executions * fix bug causing modal to close * make tags in workflow open clickable * increase workflow limit to 3 * remove success notifications * update header spacing * escape tag names * update tag and table colors * remove tags from export * build * clean up push connection dependencies * address ben's comments * revert tags optional interface * address comments * update duplicate message * build * fix eol * add one more eol * :zap: Update comment * add hover style for workflow open, fix up font weight Co-authored-by: Mutasem <mutdmour@gmail.com> Co-authored-by: Iván Ovejero <ivov.src@gmail.com> Co-authored-by: Mutasem Aldmour <4711238+mutdmour@users.noreply.github.com>
2021-05-29 11:31:21 -07:00
const workflowId = executionData.workflowData.id;
if (workflowId !== undefined && isWorkflowIdValid(workflowId)) {
fullExecutionData.workflowId = workflowId;
:sparkles: Implement Wait functionality (#1817) * refactor saving * refactor api layer to be stateless * refactor header details * set variable for menu height * clean up scss * clean up indentation * clean up dropdown impl * refactor no tags view * split away header * Fix tslint issues * Refactor tag manager * add tags to patch request * clean up scss * :zap: Refactor types to entities * fix issues * update no workflow error * clean up tagscontainer * use getters instead of state * remove imports * use custom colors * clean up tags container * clean up dropdown * clean up focusoncreate * :zap: Ignore mistaken ID in POST /workflows * :zap: Fix undefined tag ID in PATCH /workflows * :zap: Shorten response for POST /tags * remove scss mixins * clean up imports * :zap: Implement validation with class-validator * address ivan's comments * implement modals * Fix lint issues * fix disabling shortcuts * fix focus issues * fix focus issues * fix focus issues with modal * fix linting issues * use dispatch * use constants for modal keys * fix focus * fix lint issues * remove unused prop * add modal root * fix lint issues * remove unused methods * fix shortcut * remove max width * :zap: Fix duplicate entry error for pg and MySQL * update rename messaging * update order of buttons * fix firefox overflow on windows * fix dropdown height * :hammer: refactor tag crud controllers * 🧹 remove unused imports * use variable for number of items * fix dropdown spacing * :zap: Restore type to fix build * :zap: Fix post-refactor PATCH /workflows/:id * :zap: Fix PATCH /workflows/:id for zero tags * :zap: Fix usage count becoming stringified * address max's comments * fix filter spacing * fix blur bug * address most of ivan's comments * address tags type concern * remove defaults * :zap: return tag id as string * :hammer: add hooks to tag CUD operations * 🏎 simplify timestamp pruning * remove blur event * fix onblur bug * :zap: Fix fs import to fix build * address max's comments * implement responsive tag container * fix lint issues * update tag limits * address ivan's comments * remove rename, refactor header, implement new designs for save, remove responsive tag container * update styling * update styling * implement responsive tag container * implement header tags edit * implement header tags edit * fix lint issues * implement expandable input * minor fixes * minor fixes * use variable * rename save as * duplicate fixes * minor edit fixes * lint fixes * style fixes * hook up saving name * hook up tags * clean up impl * fix dirty state bug * update limit * update notification messages * on click outside * fix minor bug with count * lint fixes * handle minor edge cases * handle minor edge cases * handle minor bugs; fix firefox dropdown issue * Fix min width * apply tags only after api success * remove count fix * clean up workflow tags impl, fix tags delete bug * fix minor issue * fix minor spacing issue * disable wrap for ops * fix viewport root; save on click in dropdown * save button loading when saving name/tags * implement max width on tags container * implement cleaner create experience * disable edit while updating * codacy hex color * refactor tags container * fix clickability * fix workflow open and count * clean up structure * fix up lint issues * fix button size * increase workflow name limit for larger screen * tslint fixes * disable responsiveness for workflow modal * rename event * change min width for tags * clean up pr * address max's comments on styles * remove success toasts * add hover mode to name * minor fixes * refactor name preview * fix name input not to jiggle * finish up name input * Fix up add tags * clean up param * clean up scss * fix resizing name * fix resizing name * fix resize bug * clean up edit spacing * ignore on esc * fix input bug * focus input on clear * build * fix up add tags clickablity * remove scrollbars * move into folders * clean up multiple patch req * remove padding top from edit * update tags on enter * build * rollout blur on enter behavior * rollout esc behavior * fix tags bug when duplicating tags * move key to reload tags * update header spacing * build * update hex case * refactor workflow title * remove unusued prop * keep focus on error, fix bug on error * Fix bug with name / tags toggle on error * fix connection push bug * :spakles: Implement wait functionality * :bug: Do not delete waiting executions with prune * :zap: Improve SQLite migration to not lose execution data anymore * :zap: Make it possible to restart waiting execution via webhook * :zap: Add missing file * :bug: Some more merge fixes * :zap: Do not show error for Wait-Nodes if in time-mode * :zap: Make $executionId available in expressions * :shirt: Fix lint issue * :shirt: Fix lint issue * :shirt: Fix lint issue * :zap: Set the unlimited sleep time as a variable * :zap: Add also sleeping webhook path to config * :zap: Make it possible to retrieve restartUrl in workflow * :zap: Add authentication to Wait-Node in Webhook-Mode * :zap: Return 404 when trying to restart execution via webhook which does not support it * :sparkles: Make it possible to set absolute time on Wait-Node * :zap: Remove not needed imports * :zap: Fix description format * :sparkles: Implement missing webhook features on Wait-Node * :zap: Display webhook variable in NodeWebhooks * :zap: Include also date in displayed sleep time * :zap: Make it possible to see sleep time on node * :zap: Make sure that no executions does get executed twice * :zap: Add comment * :zap: Further improvements * :zap: Make Wait-Node easier to use * :sparkles: Add support for "notice" parameter type * Fixing wait node to work with queue, improved logging and execution view * Added support for mysql and pg * :sparkles: Add support for webhook postfix path * :sparkles: Make it possible to stop sleeping executions * :zap: Fix issue with webhook paths in not webhook mode * :zap: Remove not needed console.log * :zap: Update TODOs * :zap: Increase min time of workflow staying active to descrease possible issue with overlap * :shirt: Fix lint issue * :bug: Fix issues with webhooks * :zap: Make error message clearer * :zap: Fix issue with missing execution ID in scaling mode * Fixed execution list to correctly display waiting executins * Feature: enable webhook wait workflows to continue after specified time * Fixed linting * :zap: Improve waiting description text * :zap: Fix parameter display issue and rename * :zap: Remove comment * :zap: Do not display webhooks on Wait-Node * Changed wording from restart to resume on wait node * Fixed wording and inconsistent screen when changing resume modes * Removed dots from the descriptions * Changed docs url and renaming postfix to suffix * Changed names from sleep to wait * :zap: Apply suggestions from ben Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> * Some fixes by Ben * :zap: Remove console.logs * :zap: Fixes and improvements Co-authored-by: Mutasem <mutdmour@gmail.com> Co-authored-by: Iván Ovejero <ivov.src@gmail.com> Co-authored-by: Ben Hesseldieck <b.hesseldieck@gmail.com> Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> Co-authored-by: Omar Ajoue <krynble@gmail.com>
2021-08-21 05:11:32 -07:00
}
:sparkles: Unify execution id + Queue system (#1340) * Unify execution ID across executions * Fix indentation and improved comments * WIP: saving data after each node execution * Added on/off to save data after each step, saving initial data and retries working * Fixing lint issues * Fixing more lint issues * :sparkles: Add bull to execute workflows * :shirt: Fix lint issue * :zap: Add graceful shutdown to worker * :zap: Add loading staticData to worker * :shirt: Fix lint issue * :zap: Fix import * Changed tables metadata to add nullable to stoppedAt * Reload database on migration run * Fixed reloading database schema for sqlite by reconnecting and fixing postgres migration * Added checks to Redis and exiting process if connection is unavailable * Fixing error with new installations * Fix issue with data not being sent back to browser on manual executions with defined destination * Merging bull and unify execution id branch fixes * Main process will now get execution success from database instead of redis * Omit execution duration if execution did not stop * Fix issue with execution list displaying inconsistant information information while a workflow is running * Remove unused hooks to clarify for developers that these wont run in queue mode * Added active pooling to help recover from Redis crashes * Lint issues * Changing default polling interval to 60 seconds * Removed unnecessary attributes from bull job * :zap: Improved output on worker job start Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
2021-02-08 23:59:32 -08:00
executionId = await this.executionRepository.createNewExecution(fullExecutionData);
feat(core): Add execution runData recovery and status field (#5112) * adds ExecutionEvents view modal to ExecutionList * fix time rendering and remove wf column * checks for unfinished executions and fails them * prevent re-setting stoppedAt for execution * some cleanup / manually create rundata after crash * quicksave * remove Threads lib, log worker rewrite * cleanup comment * fix sentry destination return value * test for tests... * run tests with single worker * fix tests * remove console log * add endpoint for execution data recovery * lint cleanup and some refactoring * fix accidental recursion * remove cyclic imports * add rundata recovery to Workflowrunner * remove comments * cleanup and refactor * adds a status field to executions * setExecutionStatus on queued worker * fix onWorkflowPostExecute * set waiting from worker * get crashed status into frontend * remove comment * merge fix * cleanup * catch empty rundata in recovery * refactor IExecutionsSummary and inject nodeExecution Errors * reduce default event log size to 10mb from 100mb * add per node execution status * lint fix * merge and lint fix * phrasing change * improve preview rendering and messaging * remove debug * Improve partial rundata recovery * fix labels * fix line through * send manual rundata to ui at crash * some type and msg push fixes * improve recovered item rendering in preview * update workflowStatistics on recover * merge fix * review fixes * merge fix * notify eventbus when ui is back up * add a small timeout to make sure the UI is back up * increase reconnect timeout to 30s * adjust recover timeout and ui connection lost msg * do not stop execution in editor after x reconnects * add executionRecovered push event * fix recovered connection not green * remove reconnect toast and merge existing rundata * merge editor and recovered data for own mode
2023-02-17 01:54:07 -08:00
if (executionId === undefined) {
throw new ApplicationError('There was an issue assigning an execution id to the execution');
feat(core): Add execution runData recovery and status field (#5112) * adds ExecutionEvents view modal to ExecutionList * fix time rendering and remove wf column * checks for unfinished executions and fails them * prevent re-setting stoppedAt for execution * some cleanup / manually create rundata after crash * quicksave * remove Threads lib, log worker rewrite * cleanup comment * fix sentry destination return value * test for tests... * run tests with single worker * fix tests * remove console log * add endpoint for execution data recovery * lint cleanup and some refactoring * fix accidental recursion * remove cyclic imports * add rundata recovery to Workflowrunner * remove comments * cleanup and refactor * adds a status field to executions * setExecutionStatus on queued worker * fix onWorkflowPostExecute * set waiting from worker * get crashed status into frontend * remove comment * merge fix * cleanup * catch empty rundata in recovery * refactor IExecutionsSummary and inject nodeExecution Errors * reduce default event log size to 10mb from 100mb * add per node execution status * lint fix * merge and lint fix * phrasing change * improve preview rendering and messaging * remove debug * Improve partial rundata recovery * fix labels * fix line through * send manual rundata to ui at crash * some type and msg push fixes * improve recovered item rendering in preview * update workflowStatistics on recover * merge fix * review fixes * merge fix * notify eventbus when ui is back up * add a small timeout to make sure the UI is back up * increase reconnect timeout to 30s * adjust recover timeout and ui connection lost msg * do not stop execution in editor after x reconnects * add executionRecovered push event * fix recovered connection not green * remove reconnect toast and merge existing rundata * merge editor and recovered data for own mode
2023-02-17 01:54:07 -08:00
}
executionStatus = 'running';
:sparkles: Implement Wait functionality (#1817) * refactor saving * refactor api layer to be stateless * refactor header details * set variable for menu height * clean up scss * clean up indentation * clean up dropdown impl * refactor no tags view * split away header * Fix tslint issues * Refactor tag manager * add tags to patch request * clean up scss * :zap: Refactor types to entities * fix issues * update no workflow error * clean up tagscontainer * use getters instead of state * remove imports * use custom colors * clean up tags container * clean up dropdown * clean up focusoncreate * :zap: Ignore mistaken ID in POST /workflows * :zap: Fix undefined tag ID in PATCH /workflows * :zap: Shorten response for POST /tags * remove scss mixins * clean up imports * :zap: Implement validation with class-validator * address ivan's comments * implement modals * Fix lint issues * fix disabling shortcuts * fix focus issues * fix focus issues * fix focus issues with modal * fix linting issues * use dispatch * use constants for modal keys * fix focus * fix lint issues * remove unused prop * add modal root * fix lint issues * remove unused methods * fix shortcut * remove max width * :zap: Fix duplicate entry error for pg and MySQL * update rename messaging * update order of buttons * fix firefox overflow on windows * fix dropdown height * :hammer: refactor tag crud controllers * 🧹 remove unused imports * use variable for number of items * fix dropdown spacing * :zap: Restore type to fix build * :zap: Fix post-refactor PATCH /workflows/:id * :zap: Fix PATCH /workflows/:id for zero tags * :zap: Fix usage count becoming stringified * address max's comments * fix filter spacing * fix blur bug * address most of ivan's comments * address tags type concern * remove defaults * :zap: return tag id as string * :hammer: add hooks to tag CUD operations * 🏎 simplify timestamp pruning * remove blur event * fix onblur bug * :zap: Fix fs import to fix build * address max's comments * implement responsive tag container * fix lint issues * update tag limits * address ivan's comments * remove rename, refactor header, implement new designs for save, remove responsive tag container * update styling * update styling * implement responsive tag container * implement header tags edit * implement header tags edit * fix lint issues * implement expandable input * minor fixes * minor fixes * use variable * rename save as * duplicate fixes * minor edit fixes * lint fixes * style fixes * hook up saving name * hook up tags * clean up impl * fix dirty state bug * update limit * update notification messages * on click outside * fix minor bug with count * lint fixes * handle minor edge cases * handle minor edge cases * handle minor bugs; fix firefox dropdown issue * Fix min width * apply tags only after api success * remove count fix * clean up workflow tags impl, fix tags delete bug * fix minor issue * fix minor spacing issue * disable wrap for ops * fix viewport root; save on click in dropdown * save button loading when saving name/tags * implement max width on tags container * implement cleaner create experience * disable edit while updating * codacy hex color * refactor tags container * fix clickability * fix workflow open and count * clean up structure * fix up lint issues * fix button size * increase workflow name limit for larger screen * tslint fixes * disable responsiveness for workflow modal * rename event * change min width for tags * clean up pr * address max's comments on styles * remove success toasts * add hover mode to name * minor fixes * refactor name preview * fix name input not to jiggle * finish up name input * Fix up add tags * clean up param * clean up scss * fix resizing name * fix resizing name * fix resize bug * clean up edit spacing * ignore on esc * fix input bug * focus input on clear * build * fix up add tags clickablity * remove scrollbars * move into folders * clean up multiple patch req * remove padding top from edit * update tags on enter * build * rollout blur on enter behavior * rollout esc behavior * fix tags bug when duplicating tags * move key to reload tags * update header spacing * build * update hex case * refactor workflow title * remove unusued prop * keep focus on error, fix bug on error * Fix bug with name / tags toggle on error * fix connection push bug * :spakles: Implement wait functionality * :bug: Do not delete waiting executions with prune * :zap: Improve SQLite migration to not lose execution data anymore * :zap: Make it possible to restart waiting execution via webhook * :zap: Add missing file * :bug: Some more merge fixes * :zap: Do not show error for Wait-Nodes if in time-mode * :zap: Make $executionId available in expressions * :shirt: Fix lint issue * :shirt: Fix lint issue * :shirt: Fix lint issue * :zap: Set the unlimited sleep time as a variable * :zap: Add also sleeping webhook path to config * :zap: Make it possible to retrieve restartUrl in workflow * :zap: Add authentication to Wait-Node in Webhook-Mode * :zap: Return 404 when trying to restart execution via webhook which does not support it * :sparkles: Make it possible to set absolute time on Wait-Node * :zap: Remove not needed imports * :zap: Fix description format * :sparkles: Implement missing webhook features on Wait-Node * :zap: Display webhook variable in NodeWebhooks * :zap: Include also date in displayed sleep time * :zap: Make it possible to see sleep time on node * :zap: Make sure that no executions does get executed twice * :zap: Add comment * :zap: Further improvements * :zap: Make Wait-Node easier to use * :sparkles: Add support for "notice" parameter type * Fixing wait node to work with queue, improved logging and execution view * Added support for mysql and pg * :sparkles: Add support for webhook postfix path * :sparkles: Make it possible to stop sleeping executions * :zap: Fix issue with webhook paths in not webhook mode * :zap: Remove not needed console.log * :zap: Update TODOs * :zap: Increase min time of workflow staying active to descrease possible issue with overlap * :shirt: Fix lint issue * :bug: Fix issues with webhooks * :zap: Make error message clearer * :zap: Fix issue with missing execution ID in scaling mode * Fixed execution list to correctly display waiting executins * Feature: enable webhook wait workflows to continue after specified time * Fixed linting * :zap: Improve waiting description text * :zap: Fix parameter display issue and rename * :zap: Remove comment * :zap: Do not display webhooks on Wait-Node * Changed wording from restart to resume on wait node * Fixed wording and inconsistent screen when changing resume modes * Removed dots from the descriptions * Changed docs url and renaming postfix to suffix * Changed names from sleep to wait * :zap: Apply suggestions from ben Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> * Some fixes by Ben * :zap: Remove console.logs * :zap: Fixes and improvements Co-authored-by: Mutasem <mutdmour@gmail.com> Co-authored-by: Iván Ovejero <ivov.src@gmail.com> Co-authored-by: Ben Hesseldieck <b.hesseldieck@gmail.com> Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> Co-authored-by: Omar Ajoue <krynble@gmail.com>
2021-08-21 05:11:32 -07:00
} else {
// Is an existing execution we want to finish so update in DB
feat: Migrate integer primary keys to nanoids (#6345) * first commit for postgres migration * (not working) * sqlite migration * quicksave * fix tests * fix pg test * fix postgres * fix variables import * fix execution saving * add user settings fix * change migration to single lines * patch preferences endpoint * cleanup * improve variable import * cleanup unusued code * Update packages/cli/src/PublicApi/v1/handlers/workflows/workflows.handler.ts Co-authored-by: Omar Ajoue <krynble@gmail.com> * address review notes * fix var update/import * refactor: Separate execution data to its own table (#6323) * wip: Temporary migration process * refactor: Create boilerplate repository methods for executions * fix: Lint issues * refactor: Added search endpoint to repository * refactor: Make the execution list work again * wip: Updating how we create and update executions everywhere * fix: Lint issues and remove most of the direct access to execution model * refactor: Remove includeWorkflowData flag and fix more tests * fix: Lint issues * fix: Fixed ordering of executions for FE, removed transaction when saving execution and removed unnecessary update * refactor: Add comment about missing feature * refactor: Refactor counting executions * refactor: Add migration for other dbms and fix issues found * refactor: Fix lint issues * refactor: Remove unnecessary comment and auto inject repo to internal hooks * refactor: remove type assertion * fix: Fix broken tests * fix: Remove unnecessary import * Remove unnecessary toString() call Co-authored-by: Iván Ovejero <ivov.src@gmail.com> * fix: Address comments after review * refactor: Remove unused import * fix: Lint issues * fix: Add correct migration files --------- Co-authored-by: Iván Ovejero <ivov.src@gmail.com> * remove null values from credential export * fix: Fix an issue with queue mode where all running execution would be returned * fix: Update n8n node to allow for workflow ids with letters * set upstream on set branch * remove typo * add nodeAccess to credentials * fix unsaved run check for undefined id * fix(core): Rename version control feature to source control (#6480) * rename versionControl to sourceControl * fix source control tooltip wording --------- Co-authored-by: Romain Minaud <romain.minaud@gmail.com> * fix(editor): Pay 548 hide the set up version control button (#6485) * feat(DebugHelper Node): Fix and include in main app (#6406) * improve node a bit * fixing continueOnFail() ton contain error in json * improve pairedItem * fix random data returning object results * fix nanoId length typo * update pnpm-lock file --------- Co-authored-by: Marcus <marcus@n8n.io> * fix(editor): Remove setup source control CTA button * fix(editor): Remove setup source control CTA button --------- Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com> Co-authored-by: Marcus <marcus@n8n.io> * fix(editor): Update source control docs links (#6488) * feat(DebugHelper Node): Fix and include in main app (#6406) * improve node a bit * fixing continueOnFail() ton contain error in json * improve pairedItem * fix random data returning object results * fix nanoId length typo * update pnpm-lock file --------- Co-authored-by: Marcus <marcus@n8n.io> * feat(editor): Replace root events with event bus events (no-changelog) (#6454) * feat: replace root events with event bus events * fix: prevent cypress from replacing global with globalThis in import path * feat: remove emitter mixin * fix: replace component events with event bus * fix: fix linting issue * fix: fix breaking expression switch * chore: prettify ndv e2e suite code * fix(editor): Update source control docs links --------- Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com> Co-authored-by: Marcus <marcus@n8n.io> Co-authored-by: Alex Grozav <alex@grozav.com> * fix tag endpoint regex --------- Co-authored-by: Omar Ajoue <krynble@gmail.com> Co-authored-by: Iván Ovejero <ivov.src@gmail.com> Co-authored-by: Romain Minaud <romain.minaud@gmail.com> Co-authored-by: Csaba Tuncsik <csaba@n8n.io> Co-authored-by: Marcus <marcus@n8n.io> Co-authored-by: Alex Grozav <alex@grozav.com>
2023-06-20 10:13:18 -07:00
const execution: Pick<IExecutionDb, 'id' | 'data' | 'waitTill' | 'status'> = {
:sparkles: Implement Wait functionality (#1817) * refactor saving * refactor api layer to be stateless * refactor header details * set variable for menu height * clean up scss * clean up indentation * clean up dropdown impl * refactor no tags view * split away header * Fix tslint issues * Refactor tag manager * add tags to patch request * clean up scss * :zap: Refactor types to entities * fix issues * update no workflow error * clean up tagscontainer * use getters instead of state * remove imports * use custom colors * clean up tags container * clean up dropdown * clean up focusoncreate * :zap: Ignore mistaken ID in POST /workflows * :zap: Fix undefined tag ID in PATCH /workflows * :zap: Shorten response for POST /tags * remove scss mixins * clean up imports * :zap: Implement validation with class-validator * address ivan's comments * implement modals * Fix lint issues * fix disabling shortcuts * fix focus issues * fix focus issues * fix focus issues with modal * fix linting issues * use dispatch * use constants for modal keys * fix focus * fix lint issues * remove unused prop * add modal root * fix lint issues * remove unused methods * fix shortcut * remove max width * :zap: Fix duplicate entry error for pg and MySQL * update rename messaging * update order of buttons * fix firefox overflow on windows * fix dropdown height * :hammer: refactor tag crud controllers * 🧹 remove unused imports * use variable for number of items * fix dropdown spacing * :zap: Restore type to fix build * :zap: Fix post-refactor PATCH /workflows/:id * :zap: Fix PATCH /workflows/:id for zero tags * :zap: Fix usage count becoming stringified * address max's comments * fix filter spacing * fix blur bug * address most of ivan's comments * address tags type concern * remove defaults * :zap: return tag id as string * :hammer: add hooks to tag CUD operations * 🏎 simplify timestamp pruning * remove blur event * fix onblur bug * :zap: Fix fs import to fix build * address max's comments * implement responsive tag container * fix lint issues * update tag limits * address ivan's comments * remove rename, refactor header, implement new designs for save, remove responsive tag container * update styling * update styling * implement responsive tag container * implement header tags edit * implement header tags edit * fix lint issues * implement expandable input * minor fixes * minor fixes * use variable * rename save as * duplicate fixes * minor edit fixes * lint fixes * style fixes * hook up saving name * hook up tags * clean up impl * fix dirty state bug * update limit * update notification messages * on click outside * fix minor bug with count * lint fixes * handle minor edge cases * handle minor edge cases * handle minor bugs; fix firefox dropdown issue * Fix min width * apply tags only after api success * remove count fix * clean up workflow tags impl, fix tags delete bug * fix minor issue * fix minor spacing issue * disable wrap for ops * fix viewport root; save on click in dropdown * save button loading when saving name/tags * implement max width on tags container * implement cleaner create experience * disable edit while updating * codacy hex color * refactor tags container * fix clickability * fix workflow open and count * clean up structure * fix up lint issues * fix button size * increase workflow name limit for larger screen * tslint fixes * disable responsiveness for workflow modal * rename event * change min width for tags * clean up pr * address max's comments on styles * remove success toasts * add hover mode to name * minor fixes * refactor name preview * fix name input not to jiggle * finish up name input * Fix up add tags * clean up param * clean up scss * fix resizing name * fix resizing name * fix resize bug * clean up edit spacing * ignore on esc * fix input bug * focus input on clear * build * fix up add tags clickablity * remove scrollbars * move into folders * clean up multiple patch req * remove padding top from edit * update tags on enter * build * rollout blur on enter behavior * rollout esc behavior * fix tags bug when duplicating tags * move key to reload tags * update header spacing * build * update hex case * refactor workflow title * remove unusued prop * keep focus on error, fix bug on error * Fix bug with name / tags toggle on error * fix connection push bug * :spakles: Implement wait functionality * :bug: Do not delete waiting executions with prune * :zap: Improve SQLite migration to not lose execution data anymore * :zap: Make it possible to restart waiting execution via webhook * :zap: Add missing file * :bug: Some more merge fixes * :zap: Do not show error for Wait-Nodes if in time-mode * :zap: Make $executionId available in expressions * :shirt: Fix lint issue * :shirt: Fix lint issue * :shirt: Fix lint issue * :zap: Set the unlimited sleep time as a variable * :zap: Add also sleeping webhook path to config * :zap: Make it possible to retrieve restartUrl in workflow * :zap: Add authentication to Wait-Node in Webhook-Mode * :zap: Return 404 when trying to restart execution via webhook which does not support it * :sparkles: Make it possible to set absolute time on Wait-Node * :zap: Remove not needed imports * :zap: Fix description format * :sparkles: Implement missing webhook features on Wait-Node * :zap: Display webhook variable in NodeWebhooks * :zap: Include also date in displayed sleep time * :zap: Make it possible to see sleep time on node * :zap: Make sure that no executions does get executed twice * :zap: Add comment * :zap: Further improvements * :zap: Make Wait-Node easier to use * :sparkles: Add support for "notice" parameter type * Fixing wait node to work with queue, improved logging and execution view * Added support for mysql and pg * :sparkles: Add support for webhook postfix path * :sparkles: Make it possible to stop sleeping executions * :zap: Fix issue with webhook paths in not webhook mode * :zap: Remove not needed console.log * :zap: Update TODOs * :zap: Increase min time of workflow staying active to descrease possible issue with overlap * :shirt: Fix lint issue * :bug: Fix issues with webhooks * :zap: Make error message clearer * :zap: Fix issue with missing execution ID in scaling mode * Fixed execution list to correctly display waiting executins * Feature: enable webhook wait workflows to continue after specified time * Fixed linting * :zap: Improve waiting description text * :zap: Fix parameter display issue and rename * :zap: Remove comment * :zap: Do not display webhooks on Wait-Node * Changed wording from restart to resume on wait node * Fixed wording and inconsistent screen when changing resume modes * Removed dots from the descriptions * Changed docs url and renaming postfix to suffix * Changed names from sleep to wait * :zap: Apply suggestions from ben Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> * Some fixes by Ben * :zap: Remove console.logs * :zap: Fixes and improvements Co-authored-by: Mutasem <mutdmour@gmail.com> Co-authored-by: Iván Ovejero <ivov.src@gmail.com> Co-authored-by: Ben Hesseldieck <b.hesseldieck@gmail.com> Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> Co-authored-by: Omar Ajoue <krynble@gmail.com>
2021-08-21 05:11:32 -07:00
id: executionId,
feat: Migrate integer primary keys to nanoids (#6345) * first commit for postgres migration * (not working) * sqlite migration * quicksave * fix tests * fix pg test * fix postgres * fix variables import * fix execution saving * add user settings fix * change migration to single lines * patch preferences endpoint * cleanup * improve variable import * cleanup unusued code * Update packages/cli/src/PublicApi/v1/handlers/workflows/workflows.handler.ts Co-authored-by: Omar Ajoue <krynble@gmail.com> * address review notes * fix var update/import * refactor: Separate execution data to its own table (#6323) * wip: Temporary migration process * refactor: Create boilerplate repository methods for executions * fix: Lint issues * refactor: Added search endpoint to repository * refactor: Make the execution list work again * wip: Updating how we create and update executions everywhere * fix: Lint issues and remove most of the direct access to execution model * refactor: Remove includeWorkflowData flag and fix more tests * fix: Lint issues * fix: Fixed ordering of executions for FE, removed transaction when saving execution and removed unnecessary update * refactor: Add comment about missing feature * refactor: Refactor counting executions * refactor: Add migration for other dbms and fix issues found * refactor: Fix lint issues * refactor: Remove unnecessary comment and auto inject repo to internal hooks * refactor: remove type assertion * fix: Fix broken tests * fix: Remove unnecessary import * Remove unnecessary toString() call Co-authored-by: Iván Ovejero <ivov.src@gmail.com> * fix: Address comments after review * refactor: Remove unused import * fix: Lint issues * fix: Add correct migration files --------- Co-authored-by: Iván Ovejero <ivov.src@gmail.com> * remove null values from credential export * fix: Fix an issue with queue mode where all running execution would be returned * fix: Update n8n node to allow for workflow ids with letters * set upstream on set branch * remove typo * add nodeAccess to credentials * fix unsaved run check for undefined id * fix(core): Rename version control feature to source control (#6480) * rename versionControl to sourceControl * fix source control tooltip wording --------- Co-authored-by: Romain Minaud <romain.minaud@gmail.com> * fix(editor): Pay 548 hide the set up version control button (#6485) * feat(DebugHelper Node): Fix and include in main app (#6406) * improve node a bit * fixing continueOnFail() ton contain error in json * improve pairedItem * fix random data returning object results * fix nanoId length typo * update pnpm-lock file --------- Co-authored-by: Marcus <marcus@n8n.io> * fix(editor): Remove setup source control CTA button * fix(editor): Remove setup source control CTA button --------- Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com> Co-authored-by: Marcus <marcus@n8n.io> * fix(editor): Update source control docs links (#6488) * feat(DebugHelper Node): Fix and include in main app (#6406) * improve node a bit * fixing continueOnFail() ton contain error in json * improve pairedItem * fix random data returning object results * fix nanoId length typo * update pnpm-lock file --------- Co-authored-by: Marcus <marcus@n8n.io> * feat(editor): Replace root events with event bus events (no-changelog) (#6454) * feat: replace root events with event bus events * fix: prevent cypress from replacing global with globalThis in import path * feat: remove emitter mixin * fix: replace component events with event bus * fix: fix linting issue * fix: fix breaking expression switch * chore: prettify ndv e2e suite code * fix(editor): Update source control docs links --------- Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com> Co-authored-by: Marcus <marcus@n8n.io> Co-authored-by: Alex Grozav <alex@grozav.com> * fix tag endpoint regex --------- Co-authored-by: Omar Ajoue <krynble@gmail.com> Co-authored-by: Iván Ovejero <ivov.src@gmail.com> Co-authored-by: Romain Minaud <romain.minaud@gmail.com> Co-authored-by: Csaba Tuncsik <csaba@n8n.io> Co-authored-by: Marcus <marcus@n8n.io> Co-authored-by: Alex Grozav <alex@grozav.com>
2023-06-20 10:13:18 -07:00
data: executionData.executionData!,
:sparkles: Implement Wait functionality (#1817) * refactor saving * refactor api layer to be stateless * refactor header details * set variable for menu height * clean up scss * clean up indentation * clean up dropdown impl * refactor no tags view * split away header * Fix tslint issues * Refactor tag manager * add tags to patch request * clean up scss * :zap: Refactor types to entities * fix issues * update no workflow error * clean up tagscontainer * use getters instead of state * remove imports * use custom colors * clean up tags container * clean up dropdown * clean up focusoncreate * :zap: Ignore mistaken ID in POST /workflows * :zap: Fix undefined tag ID in PATCH /workflows * :zap: Shorten response for POST /tags * remove scss mixins * clean up imports * :zap: Implement validation with class-validator * address ivan's comments * implement modals * Fix lint issues * fix disabling shortcuts * fix focus issues * fix focus issues * fix focus issues with modal * fix linting issues * use dispatch * use constants for modal keys * fix focus * fix lint issues * remove unused prop * add modal root * fix lint issues * remove unused methods * fix shortcut * remove max width * :zap: Fix duplicate entry error for pg and MySQL * update rename messaging * update order of buttons * fix firefox overflow on windows * fix dropdown height * :hammer: refactor tag crud controllers * 🧹 remove unused imports * use variable for number of items * fix dropdown spacing * :zap: Restore type to fix build * :zap: Fix post-refactor PATCH /workflows/:id * :zap: Fix PATCH /workflows/:id for zero tags * :zap: Fix usage count becoming stringified * address max's comments * fix filter spacing * fix blur bug * address most of ivan's comments * address tags type concern * remove defaults * :zap: return tag id as string * :hammer: add hooks to tag CUD operations * 🏎 simplify timestamp pruning * remove blur event * fix onblur bug * :zap: Fix fs import to fix build * address max's comments * implement responsive tag container * fix lint issues * update tag limits * address ivan's comments * remove rename, refactor header, implement new designs for save, remove responsive tag container * update styling * update styling * implement responsive tag container * implement header tags edit * implement header tags edit * fix lint issues * implement expandable input * minor fixes * minor fixes * use variable * rename save as * duplicate fixes * minor edit fixes * lint fixes * style fixes * hook up saving name * hook up tags * clean up impl * fix dirty state bug * update limit * update notification messages * on click outside * fix minor bug with count * lint fixes * handle minor edge cases * handle minor edge cases * handle minor bugs; fix firefox dropdown issue * Fix min width * apply tags only after api success * remove count fix * clean up workflow tags impl, fix tags delete bug * fix minor issue * fix minor spacing issue * disable wrap for ops * fix viewport root; save on click in dropdown * save button loading when saving name/tags * implement max width on tags container * implement cleaner create experience * disable edit while updating * codacy hex color * refactor tags container * fix clickability * fix workflow open and count * clean up structure * fix up lint issues * fix button size * increase workflow name limit for larger screen * tslint fixes * disable responsiveness for workflow modal * rename event * change min width for tags * clean up pr * address max's comments on styles * remove success toasts * add hover mode to name * minor fixes * refactor name preview * fix name input not to jiggle * finish up name input * Fix up add tags * clean up param * clean up scss * fix resizing name * fix resizing name * fix resize bug * clean up edit spacing * ignore on esc * fix input bug * focus input on clear * build * fix up add tags clickablity * remove scrollbars * move into folders * clean up multiple patch req * remove padding top from edit * update tags on enter * build * rollout blur on enter behavior * rollout esc behavior * fix tags bug when duplicating tags * move key to reload tags * update header spacing * build * update hex case * refactor workflow title * remove unusued prop * keep focus on error, fix bug on error * Fix bug with name / tags toggle on error * fix connection push bug * :spakles: Implement wait functionality * :bug: Do not delete waiting executions with prune * :zap: Improve SQLite migration to not lose execution data anymore * :zap: Make it possible to restart waiting execution via webhook * :zap: Add missing file * :bug: Some more merge fixes * :zap: Do not show error for Wait-Nodes if in time-mode * :zap: Make $executionId available in expressions * :shirt: Fix lint issue * :shirt: Fix lint issue * :shirt: Fix lint issue * :zap: Set the unlimited sleep time as a variable * :zap: Add also sleeping webhook path to config * :zap: Make it possible to retrieve restartUrl in workflow * :zap: Add authentication to Wait-Node in Webhook-Mode * :zap: Return 404 when trying to restart execution via webhook which does not support it * :sparkles: Make it possible to set absolute time on Wait-Node * :zap: Remove not needed imports * :zap: Fix description format * :sparkles: Implement missing webhook features on Wait-Node * :zap: Display webhook variable in NodeWebhooks * :zap: Include also date in displayed sleep time * :zap: Make it possible to see sleep time on node * :zap: Make sure that no executions does get executed twice * :zap: Add comment * :zap: Further improvements * :zap: Make Wait-Node easier to use * :sparkles: Add support for "notice" parameter type * Fixing wait node to work with queue, improved logging and execution view * Added support for mysql and pg * :sparkles: Add support for webhook postfix path * :sparkles: Make it possible to stop sleeping executions * :zap: Fix issue with webhook paths in not webhook mode * :zap: Remove not needed console.log * :zap: Update TODOs * :zap: Increase min time of workflow staying active to descrease possible issue with overlap * :shirt: Fix lint issue * :bug: Fix issues with webhooks * :zap: Make error message clearer * :zap: Fix issue with missing execution ID in scaling mode * Fixed execution list to correctly display waiting executins * Feature: enable webhook wait workflows to continue after specified time * Fixed linting * :zap: Improve waiting description text * :zap: Fix parameter display issue and rename * :zap: Remove comment * :zap: Do not display webhooks on Wait-Node * Changed wording from restart to resume on wait node * Fixed wording and inconsistent screen when changing resume modes * Removed dots from the descriptions * Changed docs url and renaming postfix to suffix * Changed names from sleep to wait * :zap: Apply suggestions from ben Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> * Some fixes by Ben * :zap: Remove console.logs * :zap: Fixes and improvements Co-authored-by: Mutasem <mutdmour@gmail.com> Co-authored-by: Iván Ovejero <ivov.src@gmail.com> Co-authored-by: Ben Hesseldieck <b.hesseldieck@gmail.com> Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> Co-authored-by: Omar Ajoue <krynble@gmail.com>
2021-08-21 05:11:32 -07:00
waitTill: null,
feat(core): Add execution runData recovery and status field (#5112) * adds ExecutionEvents view modal to ExecutionList * fix time rendering and remove wf column * checks for unfinished executions and fails them * prevent re-setting stoppedAt for execution * some cleanup / manually create rundata after crash * quicksave * remove Threads lib, log worker rewrite * cleanup comment * fix sentry destination return value * test for tests... * run tests with single worker * fix tests * remove console log * add endpoint for execution data recovery * lint cleanup and some refactoring * fix accidental recursion * remove cyclic imports * add rundata recovery to Workflowrunner * remove comments * cleanup and refactor * adds a status field to executions * setExecutionStatus on queued worker * fix onWorkflowPostExecute * set waiting from worker * get crashed status into frontend * remove comment * merge fix * cleanup * catch empty rundata in recovery * refactor IExecutionsSummary and inject nodeExecution Errors * reduce default event log size to 10mb from 100mb * add per node execution status * lint fix * merge and lint fix * phrasing change * improve preview rendering and messaging * remove debug * Improve partial rundata recovery * fix labels * fix line through * send manual rundata to ui at crash * some type and msg push fixes * improve recovered item rendering in preview * update workflowStatistics on recover * merge fix * review fixes * merge fix * notify eventbus when ui is back up * add a small timeout to make sure the UI is back up * increase reconnect timeout to 30s * adjust recover timeout and ui connection lost msg * do not stop execution in editor after x reconnects * add executionRecovered push event * fix recovered connection not green * remove reconnect toast and merge existing rundata * merge editor and recovered data for own mode
2023-02-17 01:54:07 -08:00
status: executionStatus,
:sparkles: Implement Wait functionality (#1817) * refactor saving * refactor api layer to be stateless * refactor header details * set variable for menu height * clean up scss * clean up indentation * clean up dropdown impl * refactor no tags view * split away header * Fix tslint issues * Refactor tag manager * add tags to patch request * clean up scss * :zap: Refactor types to entities * fix issues * update no workflow error * clean up tagscontainer * use getters instead of state * remove imports * use custom colors * clean up tags container * clean up dropdown * clean up focusoncreate * :zap: Ignore mistaken ID in POST /workflows * :zap: Fix undefined tag ID in PATCH /workflows * :zap: Shorten response for POST /tags * remove scss mixins * clean up imports * :zap: Implement validation with class-validator * address ivan's comments * implement modals * Fix lint issues * fix disabling shortcuts * fix focus issues * fix focus issues * fix focus issues with modal * fix linting issues * use dispatch * use constants for modal keys * fix focus * fix lint issues * remove unused prop * add modal root * fix lint issues * remove unused methods * fix shortcut * remove max width * :zap: Fix duplicate entry error for pg and MySQL * update rename messaging * update order of buttons * fix firefox overflow on windows * fix dropdown height * :hammer: refactor tag crud controllers * 🧹 remove unused imports * use variable for number of items * fix dropdown spacing * :zap: Restore type to fix build * :zap: Fix post-refactor PATCH /workflows/:id * :zap: Fix PATCH /workflows/:id for zero tags * :zap: Fix usage count becoming stringified * address max's comments * fix filter spacing * fix blur bug * address most of ivan's comments * address tags type concern * remove defaults * :zap: return tag id as string * :hammer: add hooks to tag CUD operations * 🏎 simplify timestamp pruning * remove blur event * fix onblur bug * :zap: Fix fs import to fix build * address max's comments * implement responsive tag container * fix lint issues * update tag limits * address ivan's comments * remove rename, refactor header, implement new designs for save, remove responsive tag container * update styling * update styling * implement responsive tag container * implement header tags edit * implement header tags edit * fix lint issues * implement expandable input * minor fixes * minor fixes * use variable * rename save as * duplicate fixes * minor edit fixes * lint fixes * style fixes * hook up saving name * hook up tags * clean up impl * fix dirty state bug * update limit * update notification messages * on click outside * fix minor bug with count * lint fixes * handle minor edge cases * handle minor edge cases * handle minor bugs; fix firefox dropdown issue * Fix min width * apply tags only after api success * remove count fix * clean up workflow tags impl, fix tags delete bug * fix minor issue * fix minor spacing issue * disable wrap for ops * fix viewport root; save on click in dropdown * save button loading when saving name/tags * implement max width on tags container * implement cleaner create experience * disable edit while updating * codacy hex color * refactor tags container * fix clickability * fix workflow open and count * clean up structure * fix up lint issues * fix button size * increase workflow name limit for larger screen * tslint fixes * disable responsiveness for workflow modal * rename event * change min width for tags * clean up pr * address max's comments on styles * remove success toasts * add hover mode to name * minor fixes * refactor name preview * fix name input not to jiggle * finish up name input * Fix up add tags * clean up param * clean up scss * fix resizing name * fix resizing name * fix resize bug * clean up edit spacing * ignore on esc * fix input bug * focus input on clear * build * fix up add tags clickablity * remove scrollbars * move into folders * clean up multiple patch req * remove padding top from edit * update tags on enter * build * rollout blur on enter behavior * rollout esc behavior * fix tags bug when duplicating tags * move key to reload tags * update header spacing * build * update hex case * refactor workflow title * remove unusued prop * keep focus on error, fix bug on error * Fix bug with name / tags toggle on error * fix connection push bug * :spakles: Implement wait functionality * :bug: Do not delete waiting executions with prune * :zap: Improve SQLite migration to not lose execution data anymore * :zap: Make it possible to restart waiting execution via webhook * :zap: Add missing file * :bug: Some more merge fixes * :zap: Do not show error for Wait-Nodes if in time-mode * :zap: Make $executionId available in expressions * :shirt: Fix lint issue * :shirt: Fix lint issue * :shirt: Fix lint issue * :zap: Set the unlimited sleep time as a variable * :zap: Add also sleeping webhook path to config * :zap: Make it possible to retrieve restartUrl in workflow * :zap: Add authentication to Wait-Node in Webhook-Mode * :zap: Return 404 when trying to restart execution via webhook which does not support it * :sparkles: Make it possible to set absolute time on Wait-Node * :zap: Remove not needed imports * :zap: Fix description format * :sparkles: Implement missing webhook features on Wait-Node * :zap: Display webhook variable in NodeWebhooks * :zap: Include also date in displayed sleep time * :zap: Make it possible to see sleep time on node * :zap: Make sure that no executions does get executed twice * :zap: Add comment * :zap: Further improvements * :zap: Make Wait-Node easier to use * :sparkles: Add support for "notice" parameter type * Fixing wait node to work with queue, improved logging and execution view * Added support for mysql and pg * :sparkles: Add support for webhook postfix path * :sparkles: Make it possible to stop sleeping executions * :zap: Fix issue with webhook paths in not webhook mode * :zap: Remove not needed console.log * :zap: Update TODOs * :zap: Increase min time of workflow staying active to descrease possible issue with overlap * :shirt: Fix lint issue * :bug: Fix issues with webhooks * :zap: Make error message clearer * :zap: Fix issue with missing execution ID in scaling mode * Fixed execution list to correctly display waiting executins * Feature: enable webhook wait workflows to continue after specified time * Fixed linting * :zap: Improve waiting description text * :zap: Fix parameter display issue and rename * :zap: Remove comment * :zap: Do not display webhooks on Wait-Node * Changed wording from restart to resume on wait node * Fixed wording and inconsistent screen when changing resume modes * Removed dots from the descriptions * Changed docs url and renaming postfix to suffix * Changed names from sleep to wait * :zap: Apply suggestions from ben Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> * Some fixes by Ben * :zap: Remove console.logs * :zap: Fixes and improvements Co-authored-by: Mutasem <mutdmour@gmail.com> Co-authored-by: Iván Ovejero <ivov.src@gmail.com> Co-authored-by: Ben Hesseldieck <b.hesseldieck@gmail.com> Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> Co-authored-by: Omar Ajoue <krynble@gmail.com>
2021-08-21 05:11:32 -07:00
};
await this.executionRepository.updateExistingExecution(executionId, execution);
:sparkles: Implement Wait functionality (#1817) * refactor saving * refactor api layer to be stateless * refactor header details * set variable for menu height * clean up scss * clean up indentation * clean up dropdown impl * refactor no tags view * split away header * Fix tslint issues * Refactor tag manager * add tags to patch request * clean up scss * :zap: Refactor types to entities * fix issues * update no workflow error * clean up tagscontainer * use getters instead of state * remove imports * use custom colors * clean up tags container * clean up dropdown * clean up focusoncreate * :zap: Ignore mistaken ID in POST /workflows * :zap: Fix undefined tag ID in PATCH /workflows * :zap: Shorten response for POST /tags * remove scss mixins * clean up imports * :zap: Implement validation with class-validator * address ivan's comments * implement modals * Fix lint issues * fix disabling shortcuts * fix focus issues * fix focus issues * fix focus issues with modal * fix linting issues * use dispatch * use constants for modal keys * fix focus * fix lint issues * remove unused prop * add modal root * fix lint issues * remove unused methods * fix shortcut * remove max width * :zap: Fix duplicate entry error for pg and MySQL * update rename messaging * update order of buttons * fix firefox overflow on windows * fix dropdown height * :hammer: refactor tag crud controllers * 🧹 remove unused imports * use variable for number of items * fix dropdown spacing * :zap: Restore type to fix build * :zap: Fix post-refactor PATCH /workflows/:id * :zap: Fix PATCH /workflows/:id for zero tags * :zap: Fix usage count becoming stringified * address max's comments * fix filter spacing * fix blur bug * address most of ivan's comments * address tags type concern * remove defaults * :zap: return tag id as string * :hammer: add hooks to tag CUD operations * 🏎 simplify timestamp pruning * remove blur event * fix onblur bug * :zap: Fix fs import to fix build * address max's comments * implement responsive tag container * fix lint issues * update tag limits * address ivan's comments * remove rename, refactor header, implement new designs for save, remove responsive tag container * update styling * update styling * implement responsive tag container * implement header tags edit * implement header tags edit * fix lint issues * implement expandable input * minor fixes * minor fixes * use variable * rename save as * duplicate fixes * minor edit fixes * lint fixes * style fixes * hook up saving name * hook up tags * clean up impl * fix dirty state bug * update limit * update notification messages * on click outside * fix minor bug with count * lint fixes * handle minor edge cases * handle minor edge cases * handle minor bugs; fix firefox dropdown issue * Fix min width * apply tags only after api success * remove count fix * clean up workflow tags impl, fix tags delete bug * fix minor issue * fix minor spacing issue * disable wrap for ops * fix viewport root; save on click in dropdown * save button loading when saving name/tags * implement max width on tags container * implement cleaner create experience * disable edit while updating * codacy hex color * refactor tags container * fix clickability * fix workflow open and count * clean up structure * fix up lint issues * fix button size * increase workflow name limit for larger screen * tslint fixes * disable responsiveness for workflow modal * rename event * change min width for tags * clean up pr * address max's comments on styles * remove success toasts * add hover mode to name * minor fixes * refactor name preview * fix name input not to jiggle * finish up name input * Fix up add tags * clean up param * clean up scss * fix resizing name * fix resizing name * fix resize bug * clean up edit spacing * ignore on esc * fix input bug * focus input on clear * build * fix up add tags clickablity * remove scrollbars * move into folders * clean up multiple patch req * remove padding top from edit * update tags on enter * build * rollout blur on enter behavior * rollout esc behavior * fix tags bug when duplicating tags * move key to reload tags * update header spacing * build * update hex case * refactor workflow title * remove unusued prop * keep focus on error, fix bug on error * Fix bug with name / tags toggle on error * fix connection push bug * :spakles: Implement wait functionality * :bug: Do not delete waiting executions with prune * :zap: Improve SQLite migration to not lose execution data anymore * :zap: Make it possible to restart waiting execution via webhook * :zap: Add missing file * :bug: Some more merge fixes * :zap: Do not show error for Wait-Nodes if in time-mode * :zap: Make $executionId available in expressions * :shirt: Fix lint issue * :shirt: Fix lint issue * :shirt: Fix lint issue * :zap: Set the unlimited sleep time as a variable * :zap: Add also sleeping webhook path to config * :zap: Make it possible to retrieve restartUrl in workflow * :zap: Add authentication to Wait-Node in Webhook-Mode * :zap: Return 404 when trying to restart execution via webhook which does not support it * :sparkles: Make it possible to set absolute time on Wait-Node * :zap: Remove not needed imports * :zap: Fix description format * :sparkles: Implement missing webhook features on Wait-Node * :zap: Display webhook variable in NodeWebhooks * :zap: Include also date in displayed sleep time * :zap: Make it possible to see sleep time on node * :zap: Make sure that no executions does get executed twice * :zap: Add comment * :zap: Further improvements * :zap: Make Wait-Node easier to use * :sparkles: Add support for "notice" parameter type * Fixing wait node to work with queue, improved logging and execution view * Added support for mysql and pg * :sparkles: Add support for webhook postfix path * :sparkles: Make it possible to stop sleeping executions * :zap: Fix issue with webhook paths in not webhook mode * :zap: Remove not needed console.log * :zap: Update TODOs * :zap: Increase min time of workflow staying active to descrease possible issue with overlap * :shirt: Fix lint issue * :bug: Fix issues with webhooks * :zap: Make error message clearer * :zap: Fix issue with missing execution ID in scaling mode * Fixed execution list to correctly display waiting executins * Feature: enable webhook wait workflows to continue after specified time * Fixed linting * :zap: Improve waiting description text * :zap: Fix parameter display issue and rename * :zap: Remove comment * :zap: Do not display webhooks on Wait-Node * Changed wording from restart to resume on wait node * Fixed wording and inconsistent screen when changing resume modes * Removed dots from the descriptions * Changed docs url and renaming postfix to suffix * Changed names from sleep to wait * :zap: Apply suggestions from ben Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> * Some fixes by Ben * :zap: Remove console.logs * :zap: Fixes and improvements Co-authored-by: Mutasem <mutdmour@gmail.com> Co-authored-by: Iván Ovejero <ivov.src@gmail.com> Co-authored-by: Ben Hesseldieck <b.hesseldieck@gmail.com> Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com> Co-authored-by: Omar Ajoue <krynble@gmail.com>
2021-08-21 05:11:32 -07:00
}
2019-06-23 03:35:23 -07:00
this.activeExecutions[executionId] = {
executionData,
startedAt: new Date(),
2019-06-23 03:35:23 -07:00
postExecutePromises: [],
feat(core): Add execution runData recovery and status field (#5112) * adds ExecutionEvents view modal to ExecutionList * fix time rendering and remove wf column * checks for unfinished executions and fails them * prevent re-setting stoppedAt for execution * some cleanup / manually create rundata after crash * quicksave * remove Threads lib, log worker rewrite * cleanup comment * fix sentry destination return value * test for tests... * run tests with single worker * fix tests * remove console log * add endpoint for execution data recovery * lint cleanup and some refactoring * fix accidental recursion * remove cyclic imports * add rundata recovery to Workflowrunner * remove comments * cleanup and refactor * adds a status field to executions * setExecutionStatus on queued worker * fix onWorkflowPostExecute * set waiting from worker * get crashed status into frontend * remove comment * merge fix * cleanup * catch empty rundata in recovery * refactor IExecutionsSummary and inject nodeExecution Errors * reduce default event log size to 10mb from 100mb * add per node execution status * lint fix * merge and lint fix * phrasing change * improve preview rendering and messaging * remove debug * Improve partial rundata recovery * fix labels * fix line through * send manual rundata to ui at crash * some type and msg push fixes * improve recovered item rendering in preview * update workflowStatistics on recover * merge fix * review fixes * merge fix * notify eventbus when ui is back up * add a small timeout to make sure the UI is back up * increase reconnect timeout to 30s * adjust recover timeout and ui connection lost msg * do not stop execution in editor after x reconnects * add executionRecovered push event * fix recovered connection not green * remove reconnect toast and merge existing rundata * merge editor and recovered data for own mode
2023-02-17 01:54:07 -08:00
status: executionStatus,
2019-06-23 03:35:23 -07:00
};
:sparkles: Unify execution id + Queue system (#1340) * Unify execution ID across executions * Fix indentation and improved comments * WIP: saving data after each node execution * Added on/off to save data after each step, saving initial data and retries working * Fixing lint issues * Fixing more lint issues * :sparkles: Add bull to execute workflows * :shirt: Fix lint issue * :zap: Add graceful shutdown to worker * :zap: Add loading staticData to worker * :shirt: Fix lint issue * :zap: Fix import * Changed tables metadata to add nullable to stoppedAt * Reload database on migration run * Fixed reloading database schema for sqlite by reconnecting and fixing postgres migration * Added checks to Redis and exiting process if connection is unavailable * Fixing error with new installations * Fix issue with data not being sent back to browser on manual executions with defined destination * Merging bull and unify execution id branch fixes * Main process will now get execution success from database instead of redis * Omit execution duration if execution did not stop * Fix issue with execution list displaying inconsistant information information while a workflow is running * Remove unused hooks to clarify for developers that these wont run in queue mode * Added active pooling to help recover from Redis crashes * Lint issues * Changing default polling interval to 60 seconds * Removed unnecessary attributes from bull job * :zap: Improved output on worker job start Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
2021-02-08 23:59:32 -08:00
return executionId;
2019-06-23 03:35:23 -07:00
}
/**
* Attaches an execution
*/
attachWorkflowExecution(executionId: string, workflowExecution: PCancelable<IRun>) {
this.getExecution(executionId).workflowExecution = workflowExecution;
}
attachResponsePromise(
executionId: string,
responsePromise: IDeferredPromise<IExecuteResponsePromiseData>,
): void {
this.getExecution(executionId).responsePromise = responsePromise;
}
resolveResponsePromise(executionId: string, response: IExecuteResponsePromiseData): void {
const execution = this.activeExecutions[executionId];
execution?.responsePromise?.resolve(response);
}
refactor(core): Move event and telemetry handling into workers in queue mode (#7138) # Motivation In Queue mode, finished executions would cause the main instance to always pull all execution data from the database, unflatten it and then use it to send out event log events and telemetry events, as well as required returns to Respond to Webhook nodes etc. This could cause OOM errors when the data was large, since it had to be fully unpacked and transformed on the main instance’s side, using up a lot of memory (and time). This PR attempts to limit this behaviour to only happen in those required cases where the data has to be forwarded to some waiting webhook, for example. # Changes Execution data is only required in cases, where the active execution has a `postExecutePromise` attached to it. These usually forward the data to some other endpoint (e.g. a listening webhook connection). By adding a helper `getPostExecutePromiseCount()`, we can decide that in cases where there is nothing listening at all, there is no reason to pull the data on the main instance. Previously, there would always be postExecutePromises because the telemetry events were called. Now, these have been moved into the workers, which have been given the various InternalHooks calls to their hook function arrays, so they themselves issue these telemetry and event calls. This results in all event log messages to now be logged on the worker’s event log, as well as the worker’s eventbus being the one to send out the events to destinations. The main event log does…pretty much nothing. We are not logging executions on the main event log any more, because this would require all events to be replicated 1:1 from the workers to the main instance(s) (this IS possible and implemented, see the worker’s `replicateToRedisEventLogFunction` - but it is not enabled to reduce the amount of traffic over redis). Partial events in the main log could confuse the recovery process and would result in, ironically, the recovery corrupting the execution data by considering them crashed. # Refactor I have also used the opportunity to reduce duplicate code and move some of the hook functionality into `packages/cli/src/executionLifecycleHooks/shared/sharedHookFunctions.ts` in preparation for a future full refactor of the hooks
2023-09-13 22:58:15 -07:00
getPostExecutePromiseCount(executionId: string): number {
return this.activeExecutions[executionId]?.postExecutePromises.length ?? 0;
}
2019-06-23 03:35:23 -07:00
/**
* Remove an active execution
*/
remove(executionId: string, fullRunData?: IRun): void {
const execution = this.activeExecutions[executionId];
if (execution === undefined) {
2019-06-23 03:35:23 -07:00
return;
}
// Resolve all the waiting promises
for (const promise of execution.postExecutePromises) {
2019-06-23 03:35:23 -07:00
promise.resolve(fullRunData);
}
// Remove from the list of active executions
delete this.activeExecutions[executionId];
}
/**
* Forces an execution to stop
*/
async stopExecution(executionId: string): Promise<IRun | undefined> {
const execution = this.activeExecutions[executionId];
if (execution === undefined) {
2019-06-23 03:35:23 -07:00
// There is no execution running with that id
return;
}
execution.workflowExecution!.cancel();
2019-06-23 03:35:23 -07:00
return await this.getPostExecutePromise(executionId);
2019-06-23 03:35:23 -07:00
}
/**
* Returns a promise which will resolve with the data of the execution with the given id
2019-06-23 03:35:23 -07:00
*/
async getPostExecutePromise(executionId: string): Promise<IRun | undefined> {
// Create the promise which will be resolved when the execution finished
const waitPromise = await createDeferredPromise<IRun | undefined>();
this.getExecution(executionId).postExecutePromises.push(waitPromise);
return await waitPromise.promise();
2019-06-23 03:35:23 -07:00
}
/**
* Returns all the currently active executions
*/
getActiveExecutions(): IExecutionsCurrentSummary[] {
const returnData: IExecutionsCurrentSummary[] = [];
let data;
2019-06-23 03:35:23 -07:00
for (const id of Object.keys(this.activeExecutions)) {
data = this.activeExecutions[id];
2019-06-23 03:35:23 -07:00
returnData.push({
id,
retryOf: data.executionData.retryOf,
startedAt: data.startedAt,
mode: data.executionData.executionMode,
workflowId: data.executionData.workflowData.id,
feat(core): Add execution runData recovery and status field (#5112) * adds ExecutionEvents view modal to ExecutionList * fix time rendering and remove wf column * checks for unfinished executions and fails them * prevent re-setting stoppedAt for execution * some cleanup / manually create rundata after crash * quicksave * remove Threads lib, log worker rewrite * cleanup comment * fix sentry destination return value * test for tests... * run tests with single worker * fix tests * remove console log * add endpoint for execution data recovery * lint cleanup and some refactoring * fix accidental recursion * remove cyclic imports * add rundata recovery to Workflowrunner * remove comments * cleanup and refactor * adds a status field to executions * setExecutionStatus on queued worker * fix onWorkflowPostExecute * set waiting from worker * get crashed status into frontend * remove comment * merge fix * cleanup * catch empty rundata in recovery * refactor IExecutionsSummary and inject nodeExecution Errors * reduce default event log size to 10mb from 100mb * add per node execution status * lint fix * merge and lint fix * phrasing change * improve preview rendering and messaging * remove debug * Improve partial rundata recovery * fix labels * fix line through * send manual rundata to ui at crash * some type and msg push fixes * improve recovered item rendering in preview * update workflowStatistics on recover * merge fix * review fixes * merge fix * notify eventbus when ui is back up * add a small timeout to make sure the UI is back up * increase reconnect timeout to 30s * adjust recover timeout and ui connection lost msg * do not stop execution in editor after x reconnects * add executionRecovered push event * fix recovered connection not green * remove reconnect toast and merge existing rundata * merge editor and recovered data for own mode
2023-02-17 01:54:07 -08:00
status: data.status,
2019-06-23 03:35:23 -07:00
});
}
return returnData;
}
feat(core): Add execution runData recovery and status field (#5112) * adds ExecutionEvents view modal to ExecutionList * fix time rendering and remove wf column * checks for unfinished executions and fails them * prevent re-setting stoppedAt for execution * some cleanup / manually create rundata after crash * quicksave * remove Threads lib, log worker rewrite * cleanup comment * fix sentry destination return value * test for tests... * run tests with single worker * fix tests * remove console log * add endpoint for execution data recovery * lint cleanup and some refactoring * fix accidental recursion * remove cyclic imports * add rundata recovery to Workflowrunner * remove comments * cleanup and refactor * adds a status field to executions * setExecutionStatus on queued worker * fix onWorkflowPostExecute * set waiting from worker * get crashed status into frontend * remove comment * merge fix * cleanup * catch empty rundata in recovery * refactor IExecutionsSummary and inject nodeExecution Errors * reduce default event log size to 10mb from 100mb * add per node execution status * lint fix * merge and lint fix * phrasing change * improve preview rendering and messaging * remove debug * Improve partial rundata recovery * fix labels * fix line through * send manual rundata to ui at crash * some type and msg push fixes * improve recovered item rendering in preview * update workflowStatistics on recover * merge fix * review fixes * merge fix * notify eventbus when ui is back up * add a small timeout to make sure the UI is back up * increase reconnect timeout to 30s * adjust recover timeout and ui connection lost msg * do not stop execution in editor after x reconnects * add executionRecovered push event * fix recovered connection not green * remove reconnect toast and merge existing rundata * merge editor and recovered data for own mode
2023-02-17 01:54:07 -08:00
setStatus(executionId: string, status: ExecutionStatus) {
this.getExecution(executionId).status = status;
}
getStatus(executionId: string): ExecutionStatus {
return this.getExecution(executionId).status;
}
/** Wait for all active executions to finish */
async shutdown(cancelAll = false) {
let executionIds = Object.keys(this.activeExecutions);
if (cancelAll) {
const stopPromises = executionIds.map(
async (executionId) => await this.stopExecution(executionId),
feat(core): Add execution runData recovery and status field (#5112) * adds ExecutionEvents view modal to ExecutionList * fix time rendering and remove wf column * checks for unfinished executions and fails them * prevent re-setting stoppedAt for execution * some cleanup / manually create rundata after crash * quicksave * remove Threads lib, log worker rewrite * cleanup comment * fix sentry destination return value * test for tests... * run tests with single worker * fix tests * remove console log * add endpoint for execution data recovery * lint cleanup and some refactoring * fix accidental recursion * remove cyclic imports * add rundata recovery to Workflowrunner * remove comments * cleanup and refactor * adds a status field to executions * setExecutionStatus on queued worker * fix onWorkflowPostExecute * set waiting from worker * get crashed status into frontend * remove comment * merge fix * cleanup * catch empty rundata in recovery * refactor IExecutionsSummary and inject nodeExecution Errors * reduce default event log size to 10mb from 100mb * add per node execution status * lint fix * merge and lint fix * phrasing change * improve preview rendering and messaging * remove debug * Improve partial rundata recovery * fix labels * fix line through * send manual rundata to ui at crash * some type and msg push fixes * improve recovered item rendering in preview * update workflowStatistics on recover * merge fix * review fixes * merge fix * notify eventbus when ui is back up * add a small timeout to make sure the UI is back up * increase reconnect timeout to 30s * adjust recover timeout and ui connection lost msg * do not stop execution in editor after x reconnects * add executionRecovered push event * fix recovered connection not green * remove reconnect toast and merge existing rundata * merge editor and recovered data for own mode
2023-02-17 01:54:07 -08:00
);
await Promise.allSettled(stopPromises);
feat(core): Add execution runData recovery and status field (#5112) * adds ExecutionEvents view modal to ExecutionList * fix time rendering and remove wf column * checks for unfinished executions and fails them * prevent re-setting stoppedAt for execution * some cleanup / manually create rundata after crash * quicksave * remove Threads lib, log worker rewrite * cleanup comment * fix sentry destination return value * test for tests... * run tests with single worker * fix tests * remove console log * add endpoint for execution data recovery * lint cleanup and some refactoring * fix accidental recursion * remove cyclic imports * add rundata recovery to Workflowrunner * remove comments * cleanup and refactor * adds a status field to executions * setExecutionStatus on queued worker * fix onWorkflowPostExecute * set waiting from worker * get crashed status into frontend * remove comment * merge fix * cleanup * catch empty rundata in recovery * refactor IExecutionsSummary and inject nodeExecution Errors * reduce default event log size to 10mb from 100mb * add per node execution status * lint fix * merge and lint fix * phrasing change * improve preview rendering and messaging * remove debug * Improve partial rundata recovery * fix labels * fix line through * send manual rundata to ui at crash * some type and msg push fixes * improve recovered item rendering in preview * update workflowStatistics on recover * merge fix * review fixes * merge fix * notify eventbus when ui is back up * add a small timeout to make sure the UI is back up * increase reconnect timeout to 30s * adjust recover timeout and ui connection lost msg * do not stop execution in editor after x reconnects * add executionRecovered push event * fix recovered connection not green * remove reconnect toast and merge existing rundata * merge editor and recovered data for own mode
2023-02-17 01:54:07 -08:00
}
let count = 0;
while (executionIds.length !== 0) {
if (count++ % 4 === 0) {
this.logger.info(`Waiting for ${executionIds.length} active executions to finish...`);
}
await sleep(500);
executionIds = Object.keys(this.activeExecutions);
}
feat(core): Add execution runData recovery and status field (#5112) * adds ExecutionEvents view modal to ExecutionList * fix time rendering and remove wf column * checks for unfinished executions and fails them * prevent re-setting stoppedAt for execution * some cleanup / manually create rundata after crash * quicksave * remove Threads lib, log worker rewrite * cleanup comment * fix sentry destination return value * test for tests... * run tests with single worker * fix tests * remove console log * add endpoint for execution data recovery * lint cleanup and some refactoring * fix accidental recursion * remove cyclic imports * add rundata recovery to Workflowrunner * remove comments * cleanup and refactor * adds a status field to executions * setExecutionStatus on queued worker * fix onWorkflowPostExecute * set waiting from worker * get crashed status into frontend * remove comment * merge fix * cleanup * catch empty rundata in recovery * refactor IExecutionsSummary and inject nodeExecution Errors * reduce default event log size to 10mb from 100mb * add per node execution status * lint fix * merge and lint fix * phrasing change * improve preview rendering and messaging * remove debug * Improve partial rundata recovery * fix labels * fix line through * send manual rundata to ui at crash * some type and msg push fixes * improve recovered item rendering in preview * update workflowStatistics on recover * merge fix * review fixes * merge fix * notify eventbus when ui is back up * add a small timeout to make sure the UI is back up * increase reconnect timeout to 30s * adjust recover timeout and ui connection lost msg * do not stop execution in editor after x reconnects * add executionRecovered push event * fix recovered connection not green * remove reconnect toast and merge existing rundata * merge editor and recovered data for own mode
2023-02-17 01:54:07 -08:00
}
private getExecution(executionId: string): IExecutingWorkflowData {
const execution = this.activeExecutions[executionId];
if (!execution) {
throw new ApplicationError('No active execution found', { extra: { executionId } });
}
return execution;
feat(core): Add execution runData recovery and status field (#5112) * adds ExecutionEvents view modal to ExecutionList * fix time rendering and remove wf column * checks for unfinished executions and fails them * prevent re-setting stoppedAt for execution * some cleanup / manually create rundata after crash * quicksave * remove Threads lib, log worker rewrite * cleanup comment * fix sentry destination return value * test for tests... * run tests with single worker * fix tests * remove console log * add endpoint for execution data recovery * lint cleanup and some refactoring * fix accidental recursion * remove cyclic imports * add rundata recovery to Workflowrunner * remove comments * cleanup and refactor * adds a status field to executions * setExecutionStatus on queued worker * fix onWorkflowPostExecute * set waiting from worker * get crashed status into frontend * remove comment * merge fix * cleanup * catch empty rundata in recovery * refactor IExecutionsSummary and inject nodeExecution Errors * reduce default event log size to 10mb from 100mb * add per node execution status * lint fix * merge and lint fix * phrasing change * improve preview rendering and messaging * remove debug * Improve partial rundata recovery * fix labels * fix line through * send manual rundata to ui at crash * some type and msg push fixes * improve recovered item rendering in preview * update workflowStatistics on recover * merge fix * review fixes * merge fix * notify eventbus when ui is back up * add a small timeout to make sure the UI is back up * increase reconnect timeout to 30s * adjust recover timeout and ui connection lost msg * do not stop execution in editor after x reconnects * add executionRecovered push event * fix recovered connection not green * remove reconnect toast and merge existing rundata * merge editor and recovered data for own mode
2023-02-17 01:54:07 -08:00
}
2019-06-23 03:35:23 -07:00
}