Based on #7065 | Story: https://linear.app/n8n/issue/PAY-771
n8n on filesystem mode marks binary data to delete on manual execution
deletion, on unsaved execution completion, and on every execution
pruning cycle. We later prune binary data in a separate cycle via these
marker files, based on the configured TTL. In the context of introducing
an S3 client to manage binary data, the filesystem mode's mark-and-prune
setup is too tightly coupled to the general binary data management
client interface.
This PR...
- Ensures the deletion of an execution causes the deletion of any binary
data associated to it. This does away with the need for binary data TTL
and simplifies the filesystem mode's mark-and-prune setup.
- Refactors all execution deletions (including pruning) to cause soft
deletions, hard-deletes soft-deleted executions based on the existing
pruning config, and adjusts execution endpoints to filter out
soft-deleted executions. This reduces DB load, and keeps binary data
around long enough for users to access it when building workflows with
unsaved executions.
- Moves all execution pruning work from an execution lifecycle hook to
`execution.repository.ts`. This keeps related logic in a single place.
- Removes all marking logic from the binary data manager. This
simplifies the interface that the S3 client will meet.
- Adds basic sanity-check tests to pruning logic and execution deletion.
Out of scope:
- Improving existing pruning logic.
- Improving existing execution repository logic.
- Adjusting dir structure for filesystem mode.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Github issue / Community forum post (link here to close automatically):
https://github.com/n8n-io/n8n/pull/6348
---------
Co-authored-by: Giulio Andreini <g.andreini@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
- For a saved execution, we write to disk binary data and metadata.
These two are only ever deleted via `POST /executions/delete`. No marker
file, so untouched by pruning.
- For an unsaved execution, we write to disk binary data, binary data
metadata, and a marker file at `/meta`. We later delete all three during
pruning.
- The third flow is legacy. Currently, if the execution is unsaved, we
actually store it in the DB while running the workflow and immediately
after the workflow is finished during the `onWorkflowPostExecute()` hook
we delete that execution, so the second flow applies. But formerly, we
did not store unsaved executions in the DB ("ephemeral executions") and
so we needed to write a marker file at `/persistMeta` so that, if the
ephemeral execution crashed after the step where binary data was stored,
we had a way to later delete its associated dangling binary data via a
second pruning cycle, and if the ephemeral execution succeeded, then we
immediately cleaned up the marker file at `/persistMeta` during the
`onWorkflowPostExecute()` hook.
This creation and cleanup at `/persistMeta` is still happening, but this
third flow no longer has a purpose, as we now store unsaved executions
in the DB and delete them immediately after. Hence the third flow can be
removed.
fix(core): Reduce memory consumption on BinaryDataManager.init
When there are a few thousand binary data file to delete, the `deleteMarkedFiles` and `deleteMarkedPersistedFiles` methods need a lot of memory to process these files, irrespective of if these files have any data or not.
* fix(core): Make node execution order configurable, and backward-compatible
* ⚡ Also add new Merge-Node behaviour
* ⚡ Fix typo
* Fix lint issue
* update labels
* rename legacy to v0
* remove the unnecessary log
* default all new workflows to use v1 execution-order
* remove the controller changes
* clone default settings to avoid it getting modified
---------
Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
* feat(core): Change node execution order (most top-left one first)
* ⚡ Fix issue with multi-output-nodes
* ⚡ Remove not needed meta-entry in test
* fix the e2e test
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* fix: Unify expression error behavior for v1
* fix: Add `package.json` to `tsconfig.build.json`
* fix: Make `isFrontend` a constant
* fix: Use CommonJS require to read version
* fix: Use `JSON.parse()` and `fs.readFileSync()`
* feat(editor): Make WF name a link on /executions (#6354)
* make wf name a link in exec view
* link color
* make wf name a link in exec view
* link color
---------
Co-authored-by: Alex Grozav <alex@grozav.com>
* fix: Try restoring inclusions in tsconfig files
* fix: Try with copy
* refactor: Switch base branch and remove global toggle
* chore: Remove unrelated changes
* chore: Restore lockfile
* fix: Ensure all expression errors fail executions
* uncaught ExpressionErrors should not fail e2e tests
---------
Co-authored-by: romainminaud <romain.minaud@gmail.com>
Co-authored-by: Alex Grozav <alex@grozav.com>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* Remove authorization header when empty
* Import pkce
* Add OAuth2 with new grant type to Twitter
* Add pkce logic auto assign authorization code if pkce not defined
* Add pkce to ui and interfaces
* Fix scopes for Oauth2 twitter
* Deubg + pass it through header
* Add debug console, add airtable cred
* Remove all console.logs, make PKCE in th body only when it exists
* Remove invalid character ~
* Remove more console.logs
* remove body inside query
* Remove useless grantype check
* Hide oauth2 twitter waiting for overhaul
* Remove redundant header removal
* Remove more console.logs
* Add comment for code verifier
* Remove uneeded scopes
* Restore client id in callback
* Revert "Add OAuth2 with new grant type to Twitter"
This reverts commit 1c3b331aa1.
* Remove oauth2 from twitter
* Remove properties linked to oauth2
* Fix lodash imports
* remove redundant check
* remove redundant codeVerifier
* patch pkce-challenge to avoid generating `code_verifier` with `~`
* store `codeVerifier` on the DB like `csrfSecret`
* remove unrelated changes
---------
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* 🔧 Adjust base ESLint config
* 🔧 Adjust `lint` and `lintfix` in `nodes-base`
* 🔧 Include `test` and `utils` in `nodes-base`
* 📘 Convert JS tests to TS
* 👕 Apply lintfixes
* fix(core): Validate customData keys and values
Throws errors in manual mode and ignores and logs values in production
* fix: validate customData key characters
* refactor: review changes
* fix: logger not initialised for metadata tests
* fix: allow numbers for values
* wip: workflow execution filtering
* fix: import type failing to build
* fix: remove console.logs
* feat: execution metadata migrations
* fix(editor): Move global executions filter to its own component
* fix(editor): Using the same filter component in workflow level
* fix(editor): a small housekeeping
* checking workflowId in filter applied
* fix(editor): update filter after resolving merge conflicts
* fix(editor): unify empy filter status
* feat(editor): add datetime picker to filter
* feat(editor): add meta fields
* fix: fix button override in datepicker panel
* feat(editor): add filter metadata
* feat(core): add 'startedBefore' execution filter prop
* feat(core): add 'tags' execution query filter
* Revert "feat(core): add 'tags' execution query filter"
This reverts commit a7b968081c.
* feat(editor): add translations and tooltip and counting selected filter props
* fix(editor): fix label layouts
* fix(editor): update custom data docs link
* fix(editor): update custom data tooltip position
* fix(editor): update tooltip text
* refactor: Ignore metadata if not enabled by license
* fix(editor): Add paywall states to advanced execution filter
* refactor: Save custom data also for worker mode
* fix: Remove duplicate migration name from list
* fix(editor): Reducing filter complexity and add debounce to text inputs
* fix(editor): Remove unused import, add comment
* fix(editor): simplify event listener
* fix: Prevent error when there are running executions
* test(editor): Add advanced execution filter basic unit test
* test(editor): Add advanced execution filter state change unit test
* fix: Small lint issue
* feat: Add indices to speed up queries
* feat: add customData limits
* refactor: put metadata save in transaction
* chore: remove unneed comment
* test: add tests for execution metadata
* fix(editor): Fixes after merge conflict
* fix(editor): Remove unused import
* wordings and ui fixes
* fix(editor): type fixes
* feat: add code node autocompletions for customData
* fix: Prevent transaction issues and ambiguous ID in sql clauses
* fix(editor): Suppress requesting current executions if metadata is used in filter (#5739)
* fix(editor): Suppress requesting current executions if metadata is used in filter
* fix(editor): Fix arrows for select in popover
* refactor: Improve performance by correcting database indices
* fix: Lint issue
* test: Fix broken test
* fix: Broken test
* test: add call data check for saveExecutionMetadata test
---------
Co-authored-by: Valya Bullions <valya@n8n.io>
Co-authored-by: Alex Grozav <alex@grozav.com>
Co-authored-by: Omar Ajoue <krynble@gmail.com>
Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
* ✨ Change from binary buffer to binary streaming
* Remove console.logs
* Import Readable from the correct lib
* stream response
* parametersToKeyValue doesn't need to be async anymore
* Fix bodyParameter reduce method
* parametersToKeyValue doesn't need to be async anymore
* handle streaming responses
* send `Content-Length` and `Content-Type` on binary requests
* Add new helper function for binary data
* Add binary function to helpers interface
* Fix bug in error handler
* Fix issue with wrongfully assigned headers to body
* Fix test workflow
* Remove console.logs
* Remove unnecsessary type
* Remove concat dependency already imported in workflow package
* Update pnpm-lock file
* Small fixes, asyncronous error message
* reset the lockfile
* Remove buffer check and simplify error handling
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* add typedi
* convert ActiveWorkflowRunner into an injectable service
* convert ExternalHooks into an injectable service
* convert InternalHooks into an injectable service
* convert LoadNodesAndCredentials into an injectable service
* convert NodeTypes and CredentialTypes into an injectable service
* convert ActiveExecutions into an injectable service
* convert WaitTracker into an injectable service
* convert Push into an injectable service
* convert ActiveWebhooks and TestWebhooks into an injectable services
* handle circular references, and log errors when a circular dependency is found
* 🚀 Release 0.216.1
* fix(core): Do not allow arbitrary path traversal in the credential-translation endpoint (#5522)
* fix(core): Do not allow arbitrary path traversal in BinaryDataManager (#5523)
* fix(core): User update endpoint should only allow updating email, firstName, and lastName (#5526)
* fix(core): Do not explicitly bypass auth on urls containing `.svg` (#5525)
* 📚 Update CHANGELOG.md
---------
Co-authored-by: janober <janober@users.noreply.github.com>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <netroy@users.noreply.github.com>
Co-authored-by: Jan Oberhauser <jan.oberhauser@gmail.com>
* adds ExecutionEvents view modal to ExecutionList
* fix time rendering and remove wf column
* checks for unfinished executions and fails them
* prevent re-setting stoppedAt for execution
* some cleanup / manually create rundata after crash
* quicksave
* remove Threads lib, log worker rewrite
* cleanup comment
* fix sentry destination return value
* test for tests...
* run tests with single worker
* fix tests
* remove console log
* add endpoint for execution data recovery
* lint cleanup and some refactoring
* fix accidental recursion
* remove cyclic imports
* add rundata recovery to Workflowrunner
* remove comments
* cleanup and refactor
* adds a status field to executions
* setExecutionStatus on queued worker
* fix onWorkflowPostExecute
* set waiting from worker
* get crashed status into frontend
* remove comment
* merge fix
* cleanup
* catch empty rundata in recovery
* refactor IExecutionsSummary and inject nodeExecution Errors
* reduce default event log size to 10mb from 100mb
* add per node execution status
* lint fix
* merge and lint fix
* phrasing change
* improve preview rendering and messaging
* remove debug
* Improve partial rundata recovery
* fix labels
* fix line through
* send manual rundata to ui at crash
* some type and msg push fixes
* improve recovered item rendering in preview
* update workflowStatistics on recover
* merge fix
* review fixes
* merge fix
* notify eventbus when ui is back up
* add a small timeout to make sure the UI is back up
* increase reconnect timeout to 30s
* adjust recover timeout and ui connection lost msg
* do not stop execution in editor after x reconnects
* add executionRecovered push event
* fix recovered connection not green
* remove reconnect toast and merge existing rundata
* merge editor and recovered data for own mode
This fixes the following issues:
* After a community node is installed, we were not calling `postProcessLoaders`, which was causing a bunch of unexpected behaviors, and sometimes even crashes. This was only happening in the session where the package was installed. After a crash, the restarted service was working without these issues.
* After a community node is installed, the icon for the nodes and credentials were missing in the UI, as we were creating one icons route per installed package at startup, and this did not handle newly installed packages. restarting the service fixes this issue as well.
Fixes https://community.n8n.io/t/showing-weird-count-on-community-nodes/23035
* feat(core): Fix populating of node custom api call options
* lint fixes
* Adress PR comments
* Add e2e test and only inject custom API options for latest version
* Make sure to injectCustomApiCallOption for the latest version of node
* feat(cli): Move apiCallOption injection to LoadNodesAndCredentials and add e2e tests to check for custom nodes credentials
* Load nodes and credentials fixtures from a single place
* Console warning if credential is invalid during customApiOptions injection
* feat(core): Fix populating of node custom api call options
* lint fixes
* Adress PR comments
* Add e2e test and only inject custom API options for latest version
* Make sure to injectCustomApiCallOption for the latest version of node
* 🐛 Fix bug on continueOnFail with pairedItem
* 🐛 Fix pairedItem continueOnFail for the Set Node
* 🎨 Add the ExpressionExtensionError back
* ⚡ import fix
---------
Co-authored-by: Michael Kret <michael.k@radency.com>
* use streams to upload files to google drive
* use streams to download files from google drive
* use resumable uploads api for google drive
* avoid dangling promises, and reduce memory usage in error logging
* ✨ Create rule `no-unneeded-backticks`
* 👕 Enable rule
* ⚡ Run rule on `cli`
* ⚡ Run rule on `core`
* ⚡ Run rule on `workflow`
* ⚡ Rule rule on `design-system`
* ⚡ Run rule on `node-dev`
* ⚡ Run rule on `editor-ui`
* ⚡ Run rule on `nodes-base`
* Setup stricter linting for typescript
* make `import/no-unresolved` an error everywhere
* use prettier to format `.vscode/settings.default.json`
* address PR comments
* stringify objects before encoding them objects in MoveBinaryData
* add fileSize and fileType on MoveBinaryData converted data
* show `view` option for text files as well
* improve how JSON binary data is shown in the UI
fix: stop using prefixed core modules from nodejs (no-changelog)
Starting node 18, there are some core modules that only work with the `node:` prefix, like `node:test`. It's very likely that nodejs will switch to prefix-only core modules in the next versions as un-prefixed core modules are susceptible to supply-chain attacks.
* Mark binary data to be deleted when pruning executions
* eslint
* make pruneExecutionData async
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
* use user-folder override consistently everywhere, including for the `.cache` folder
* use consistent config for e2e tesing, skipping config loading from env and config files
* simplify all the cypress commands, and run all e2e tests on master
* upgrade ts-node
* move tslint and typescript to a single place
* source-map-support should be loaded in the `n8n` bin script, and not in core
* upgrade jest
* Support only node.js 14, 16, or 18
* use resource locator component for database -> get (Notion V1/V2)
* getDatabases search function for V1/V2 with url
* updated database get list placeholder
* get database RLC by url - regex support optional workspace domain names
* fixed linting error
* listSearch getDatabases support filter query
* support extractValue in getCurrentNodeParameter for RLC
* RLC for database page create/getAll operation
* RLC for database get operation support "By ID" with optional v param.
* use RLC in append blocks operation
* use RLC in NotionTrigger.nodes.ts
* removed unused loadOptions getDatabases
* support database RLC in createPage/createDbPage operation
* page create operation use RLC for parent page param
* page archive operation use RLC for page param
* removed unused imports
* fixed missing extractPageId in NotionV1.node.ts
* database page get operation use RLC for page param
* database page update operation use RLC for page param
* block getAll children operation use RLC for page param
* block append operation use RLC for block param
* support databaseId with optional '-' characters
* support blockId with optional '-' characters
* support pageId with optional '-' characters
* improved RLC descriptions and hints
* NotionTrigger node support databseId with optional '-' characters
* stricter RLC by ID regex rules for uuids
* stricter RLC by URL regex rules for uuids
* stricter RLC by ID regex rules for uuids (support max length)
* RLC regex from URL allow both http and https
* RLC by ID only allow uuid v4 with optional dash
* removed RLC from URL hint "Use Notion's copy link..."
* RLC from URL only allow uuid v4
* DB Status Column: Support Simplify Properties
* Notion Credentials: Support custom Notion-Version header
Use latest Notion-Version 2022-02-22 if not set
* DB Status Column: Support DB Page Create/Update
* DB Status Column: Support DB Page GetMany Filters
* removed unused paginationToken args
* Database Get: RLC by URL improve validation error message
* ci: start supporting node.js 18
Node.js 18 becomes the active LTS on 2022-10-25, as Node.js 16 goes into maintenance mode.
https://github.com/nodejs/Release#release-schedule
* continue using node 16 for production images
* node 16 for debian and base images as well for now
* feat(cli): Setup error tracking using Sentry
* make error reporting available in the workflows package
* address some of the PR comments
* create a ErrorReporterProxy like LoggerProxy
* remove the `captureError` helper. use ErrorReporterProxy directly
* fix linting issues
* remove ErrorReporterProxy warnings in tests
* check for NODE_ENV === 'production' instead
* IErrorReporter -> ErrorReporter
* ErrorReporterProxy.getInstance() -> ErrorReporter
* allow capturing stacks in warnings as well
* make n8n debugging consistent with `npm start`
* IReportingOptions -> ReportingOptions
* use consistent signature for `error` and `warn`
* use Logger instead of console.log
* 📘 Clear all `@ts-ignore` comments from workflow package
* 👕 Default to error with package-level overrides
* refactor(core): clear all `@ts-ignore` comments from core package (#4473)
👕 Clear all `@ts-ignore` comments from core package
* ✏️ Update comment
* perf: update deepCopy
* fix: using deepCopy in core and cli packages
* fix: using deepCopy in editor
* chore: formatting
* fix: some micro optimisation in deepCopy
* feat: base structure for whatsapp node with credentials
* feat: messages operation
* feat: create generic api call with credentials and test first operation
* fix: add missing template params
* fix: language code for template
* feat: media type and start of template components
* fix: remove provider name from media type
* lintfix
* fix: format
* feat: media operations w/o upload media type
* ♻️ Convert WhatsApp Business node to declarative style
* 🐛 form data not being sent with boundary in header
* ✨ add media operations to WhatsApp
* ✨ add credentials test to WhatsApp credentials
* ♻️ move preview url to optional collection in whatsapp message
* ♻️ renamed media operations in whatsapp node
* :refactor: move media file name to optional fields in whatsapp node
* ✨ add upload from n8n for whatsapp node message resource
* 🔥 remove other template component types in whatsapp node
* :speech_bubble: add specialised text for media types in WhatsApp node
* ⚡ Load dinamically phone number and template name
* ⚡ Add action property to all operations
* 🔥 Remove unnecessary imports
* ⚡ Use getBinaryDataBuffer helper
* ⚡ Add components property
* ✨ send components for whatsapp templates and template language
* 🏷️ fix WhatsApp node message function types
* 🏷️ fix any in whatsapp message functions
* 🔥 remove unused import
* ⚡ Improvements
* ⚡ Add send location
* ⚡ Add send contact
* ⚡ Small improvement
* ♻️ changes for review
* 🐛 fix presend error
* ♻️ change lat/long to numbers with proper clamping
* fix: bad merge
* refactor: changes for review
* update package-lock.json
* update package.-lock.json
* update
Co-authored-by: cxgarcia <schlaubitzcristobal@gmail.com>
Co-authored-by: ricardo <ricardoespinoza105@gmail.com>
* ⚡ Remove duplicate and old string
* ⚡ Add telemetry
* ⚡ Futher improvements
* ⚡ Change error message and display only name of last parameter
* 👕 Fix lint issue
* ⚡ Remove not needed comments
* ⚡ Rename properties, add new ones and improve error messages
* ⚡ Add support for $execution, $prevNode and make it possible to use proxies as object
* ⚡ Some small improvements
* 🐛 Fix error message
* ⚡ Improve some error messages
* ⚡ Change resumeUrl variable and display in editor
* ⚡ Fix and extend tests
* ⚡ Multiple pairedItem improvements
* ⚡ Display "More Info" link with error messages if user can fix issue
* ⚡ Display different errors in Function Nodes
* ✨ Added resource locator interfaces to `n8n-workflow` package
* ✅ Updating Trello node to use resource locator property type
* ✨ Added resource locator prop to Delete Board` Trello operation
* ✔️ Fiixing linting errors in Trello node
* ✨ Added list mode to Trello test node
* ⚡ Updating resource locator modes interface
* ⚡ Updating Trello test node validation messages and placeholders
* N8N-4175 resource locator component (#3812)
* ✨ Implemented initial version of resource locator component
* ✨ Implemented front-end validation for resource locator component. Improved responsiveness. Minor refactoring.
* ⚡ Setting resource locator default state to list. Updating hover states and expand icon.
* 🔨 Moving resource locator component to `ParameterInput` from `ParameterInputFull
* 🔨 Moving `ResourceLocator` to a separate Vue component
* 🔨 Implementing expression and drag'n'drop support in ResourceLocator` component
* 🔨 Cleaning up `ResourceLocator` component code
* ✨ Implemented resource locator selected mode persistance
* 💄 Minor refactoring and fixes in `ResourceLocator`
* 🔨 Updating `ResourceLocator` front-end validation logic
* ⚡ Saving resource locator mode in node parameters
* 💄 Updating the `ResourceLocator` component based on the design review
* 🐛 Fixing resource locator mode parameters handling when loading node parameter values on front-end
* 💄 Removing leftover unused CSS
* ⚡ Updating interfaces to support resource locator value types
* ⚡ Updating `ResourceLocator` component to work with object parameter values
* 🔨 Cleaning up `ResourceLocator` and related components code
* ⚡ Preventing `DraggableTarget` to be sticky if disabled
* 🐛 Fixing a bug with resource locator value parameter
* 👌 Adding new type alias for all possible node parameter value types
* 👌 Updating `ResourceLocator` and related components based on PR review feedback
* ⚡ Adding disabled mode to `ResourceLocator` component, fixing expression handling, minor refactoring.
* 💄 Updating disabled state styling in `ResourceLocator` component
* ⚡ Setting correct default value for test node and removing unnecessary logic
* 💄 Added regex URL validation to Trello test node
* ✨ Updating Trello test node with another (list mode only) test case
* ✔️ Fixing linting error in Trello node
* 🔨 Removing hardcoded custom modes and modes order
* Add value extractor to routing node (#3777)
* ✨ add value extractor to routing node
* ✨ add value extractor to property modes
* 🔊 improve error logging for value extractor
* 🔥 remove old extractValue methods from RoutingNode
* ⚡ extractValue inside getNodeParameter
* 🔥 remove extract value test from RoutingNode
* ✨ make value extraction optional
* 🥅 move extract value so proper error messages are sent
* 🚨 readd accidentally removed eslint-disable
* ✨ add resource locator support extractValue
* 🚨 remove unused import
* 🐛 fix getting value of resource locator
* 💄 Updating resource locator component styling and handling reset value action
* ✨ create v2 of Trello node for resource locator
* 💄 Updating ResourceLocator droppable & activeDrop classes and removing input padding-right
* ⚡ Updating Trello test node with single-mode test case
* ⚡ Updating field names in Trello node to avoid name clash
* 💄 Updating test Trello node mode order and board:update parameter name
* 💄 Updating test node parameter names and display options
* List mode search endpoint (#3936)
* 🚧 super basic version of the search endpoint
This version is built using a hacked up version of the Google Drive
node. I need to properly create a v2 for it but it's does work.
* 🚧 fixed up type errors and return urls
* ✨ add v3 of Google Drive node with RLC
* ✨ add RLC to Google Drive Shared Drive operations
* ♻️ address some small changes requested in review
* 🐛 move list search out of /nodes/ and add check for required param
* ✨ google drive folder search
* ✨ google drive search sort by name
* ✨ add searchable flag for RLC
* ✏️ fix google drive wording for v3
* Trello and Airtable search backend (#3974)
* ✨ add search to Trello boards
* ✨ add RLC to Trello cards
* ♻️ use new versioning system for Trello v2
* 🐛 move list search out of /nodes/ and add check for required param
* ✨ re-add trello search methods
* 🥅 throw error if RLC search isn't sent a method name
This will likely be removed when the declarative style of search has
been added.
* ✨ add requires filter field to RLC search
* ✨ add searchable flag to Trello searches
* ✨ add RLC for cardId and boardId on all operations
* ✨ add ID and URL RLC to Airtable
* N8 n 4179 resource locator list mode (#3933)
* ✨ Implemented initial version of list mode dropdown
* ✨ Handling mode switching and expression support in list mode
* 🔨 Removing `sortedModes` references
* ⚡ Fixing list mode UI after latest mege
* 💄 Updating padding-right for input fields with suffix slots
* ✨ Minor fixes to validation, mode switching logic and styling
* update error
* 2 or more regex
* update regex to be more strict
* remove expr colors
* update hint
* 🚧 super basic version of the search endpoint
This version is built using a hacked up version of the Google Drive
node. I need to properly create a v2 for it but it's does work.
* 🚧 fixed up type errors and return urls
* begin list impl
* ✨ add v3 of Google Drive node with RLC
* fix ts issue
* introduce dropdown
* add more behavior
* update design
* show search
* add filtering
* push up selected
* add keyboard nav
* add loading
* add caching
* remove console
* fix build issues
* add debounce
* fix click
* keep event on focus
* fix input size bug
* add resource locator type
* update type
* update interface
* update resource locator types
* ✨ add search to Trello boards
* ✨ add RLC to Google Drive Shared Drive operations
* update
* update name
* add package
* use stringify pckg
* handle long vals
* fix bug in url id modes
* remove console log
* add lazy loading
* add lazy loading on filtering
* clean up
* make search clearable
* add error state
* ✨ add RLC to Trello cards
* ♻️ address some small changes requested in review
* ♻️ use new versioning system for Trello v2
* refactor a bit
* fix how loading happens
* clear after blur
* update api
* comment out test code
* update api
* relaod in case of error
* update endpoint
* 🐛 move list search out of /nodes/ and add check for required param
* 🐛 move list search out of /nodes/ and add check for required param
* update req handling
* update endpoint
* ✨ re-add trello search methods
* 🥅 throw error if RLC search isn't sent a method name
This will likely be removed when the declarative style of search has
been added.
* get api to work
* update scroll handling
* ✨ google drive folder search
* ✨ add requires filter field to RLC search
* ✨ google drive search sort by name
* remove console
* ✨ add searchable flag for RLC
* ✨ add searchable flag to Trello searches
* update searchable
* ✨ add RLC for cardId and boardId on all operations
* ✨ add ID and URL RLC to Airtable
* fix up search
* remove extra padding
* add link button
* update popper pos
* format
* fix formating
* update mode change
* add name urls
* update regex and errors
* upate error
* update errors
* update airtable regex
* update trello regex rules
* udpate param name
* update
* update param
* update param
* update drive node
* update params
* add keyboard nav
* fix bug
* update airtable default mode
* fix default value issue
* hide long selected value
* update duplicate reqs
* update node
* clean up impl
* dedupe resources
* fix up nv
* resort params
* update icon
* set placeholders
* default to id mode
* add telemetry
* add refresh opt
* clean up tmp val
* revert test change
* make placeholder optional
* update validation
* remove description as param hint
* support more general values
* fix links on long names
* update resource item styles
* update pos
* update icon color
* update link alt
* check if required
* move validation to workflow
* update naming
* only show warning at param level
* show right border on focus
* fix hover on all item
* fix long names bug
* fix expr bug
* add expr
* update legacy mode
* fix up impl
* clean up node types
* clean up types
* remove unnessary type
* clean up types
* clean up types
* clean up types
* clea n up localizaiton
* remove unused key
* clean up helpers
* clean up paraminput
* clean up paraminputfull
* refactor into one loop
* update component
* update class names
* update prop types
* update name cases
* update casing
* clean up classes
* clean up resource locator
* update drop handling
* update mode
* add url for link mode
* clear value by default
* add placeholder
* remove legacy hint
* handle expr in legacy
* fix typos
* revert padding change
* fix up spacing
* update to link component
* support urls for id
* fix replacement
* build
Co-authored-by: Milorad Filipovic <milorad@n8n.io>
Co-authored-by: Valya Bullions <valya@n8n.io>
* refactor: Resource locator review changes (#4109)
* ✨ Implemented initial version of list mode dropdown
* ✨ Handling mode switching and expression support in list mode
* 🔨 Removing `sortedModes` references
* ⚡ Fixing list mode UI after latest mege
* 💄 Updating padding-right for input fields with suffix slots
* ✨ Minor fixes to validation, mode switching logic and styling
* update error
* 2 or more regex
* update regex to be more strict
* remove expr colors
* update hint
* 🚧 super basic version of the search endpoint
This version is built using a hacked up version of the Google Drive
node. I need to properly create a v2 for it but it's does work.
* 🚧 fixed up type errors and return urls
* begin list impl
* ✨ add v3 of Google Drive node with RLC
* fix ts issue
* introduce dropdown
* add more behavior
* update design
* show search
* add filtering
* push up selected
* add keyboard nav
* add loading
* add caching
* remove console
* fix build issues
* add debounce
* fix click
* keep event on focus
* fix input size bug
* add resource locator type
* update type
* update interface
* update resource locator types
* ✨ add search to Trello boards
* ✨ add RLC to Google Drive Shared Drive operations
* update
* update name
* add package
* use stringify pckg
* handle long vals
* fix bug in url id modes
* remove console log
* add lazy loading
* add lazy loading on filtering
* clean up
* make search clearable
* add error state
* ✨ add RLC to Trello cards
* ♻️ address some small changes requested in review
* ♻️ use new versioning system for Trello v2
* refactor a bit
* fix how loading happens
* clear after blur
* update api
* comment out test code
* update api
* relaod in case of error
* update endpoint
* 🐛 move list search out of /nodes/ and add check for required param
* 🐛 move list search out of /nodes/ and add check for required param
* update req handling
* update endpoint
* ✨ re-add trello search methods
* 🥅 throw error if RLC search isn't sent a method name
This will likely be removed when the declarative style of search has
been added.
* get api to work
* update scroll handling
* ✨ google drive folder search
* ✨ add requires filter field to RLC search
* ✨ google drive search sort by name
* remove console
* ✨ add searchable flag for RLC
* ✨ add searchable flag to Trello searches
* update searchable
* ✨ add RLC for cardId and boardId on all operations
* ✨ add ID and URL RLC to Airtable
* fix up search
* remove extra padding
* add link button
* update popper pos
* format
* fix formating
* update mode change
* add name urls
* update regex and errors
* upate error
* update errors
* update airtable regex
* update trello regex rules
* udpate param name
* update
* update param
* update param
* update drive node
* update params
* add keyboard nav
* fix bug
* update airtable default mode
* fix default value issue
* hide long selected value
* update duplicate reqs
* update node
* clean up impl
* dedupe resources
* fix up nv
* resort params
* update icon
* set placeholders
* default to id mode
* add telemetry
* add refresh opt
* clean up tmp val
* revert test change
* make placeholder optional
* update validation
* remove description as param hint
* support more general values
* fix links on long names
* update resource item styles
* update pos
* update icon color
* update link alt
* check if required
* move validation to workflow
* update naming
* only show warning at param level
* show right border on focus
* fix hover on all item
* fix long names bug
* ♻️ refactor extractValue to allow multiple props with same name
* ♻️ use correct import for displayParameterPath
* fix expr bug
* add expr
* update legacy mode
* fix up impl
* clean up node types
* clean up types
* ♻️ remove new version of google drive node
* ♻️ removed versioned Trello node for RLC
* remove unnessary type
* ♻️ remove versioned Airtable not for RLC
* clean up types
* clean up types
* clean up types
* clea n up localizaiton
* remove unused key
* clean up helpers
* clean up paraminput
* clean up paraminputfull
* refactor into one loop
* update component
* update class names
* update prop types
* update name cases
* update casing
* clean up classes
* 💬 updated RLC URL regex error wording
* clean up resource locator
* update drop handling
* update mode
* 💬 reword value extractor errors
* 🚨 remove unneeded eslint ignores for RLC modes
* 💬 update Trello 400 error message
* 🚨 re-add removed types in editor-ui
Also ts-ignore something that was clean up in another commit. I've added
a comment to fix after someone else can look at it.
* 💬 remove hints from Google Drive RLCs
* 🥅 rethrow correct errors in Trello node
* ✨ add url for id mode on Google Drive
* 🔥 remove unused Google Drive file
* 🔊 change console.error to use logger instead
* 🔀 fix bad merges
* ♻️ small changes from review
* ♻️ remove ts-ignore
Co-authored-by: Milorad Filipovic <milorad@n8n.io>
Co-authored-by: Mutasem <mutdmour@gmail.com>
* fix build
* update tests
* fix bug with credential card
* update popover component
* fix expressions url
* fix type issue
* format
* update alt
* fix lint issues
* fix eslint issues
Co-authored-by: Milorad Filipovic <milorad@n8n.io>
Co-authored-by: Milorad FIlipović <miloradfilipovic19@gmail.com>
Co-authored-by: Valya <68596159+valya@users.noreply.github.com>
Co-authored-by: Valya Bullions <valya@n8n.io>
* 🎉 starting feature development
* ✨ sharing/unsharing a credential (#3601)
* 🎉 initial design
* ✨ sharing/unsharing of credentials
* ✅ add tests for EE credentials controller
* 💪 implement review comments
* 🛠 refactor agent creation and credential role locking
* 👕 linting adjustments (#3691)
* 👕 Adjust rule `naming-convention`
* 👕 Fix `naming-convention` config value
* 👕 Disregard casing for EE-prefixed vars
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* 🛠 refactor authAgents in tests (#3725)
* 🛠 refactor authAgent
* 🛠 refactor authAgent
* 🛠 refactor authAgent
* 🛠 refactor authAgent
* 👕 fix ts issue
* 🐘 add migration for mysql and postgres + add AuthAgent type
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* ⚡️ refactor existing credentials routes (#3672)
* 🎉 initial design
* ✨ sharing/unsharing of credentials
* ♻️ split credential update route into controller and service
* 🔥 remove credentials test that is no longer applicable
* ♻️ split credential creation route into controller and service
* ♻️ split single credential get
* ♻️ split delete credentials route
* ♻️ split get all credentials route
* 🔥 remove unused imports in credentials contoller
* 🔥 remove console.log
* :refactor: changes to credentials controller and service from review
- removed credentials from service function names
- made relations list optional
- put allowGlobalOwner in options objects
- check length of relations array so join doesn't happen if empty
- update some comments to further explain rationale
- remove unneeded `Object.assign`
- remove non-null assertion from test
* ♻️ move filtered credentials selected fields to variable
* ♻️ remove unneeded merges in credentials service
Co-authored-by: Ben Hesseldieck <b.hesseldieck@gmail.com>
Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com>
* ✅ fix test
* 🐛 fix imports
* 👕 fix lint issue
* User Management: switch over to decorators to define routes (#3827)
* Add permissions details to credentials for User Management (#3863)
* ⚡ Open `GET /users`
* ⚡ Add permissions to cred service
* 🚚 Rename method
* ⚡ Refactor cred controller
* 🧪 Adjust test
* ✏️ Improve comment
* ✏️ Improve another comment
* ⚡ Account for multiple sharings
* 🐛 Fix access when user is editor
* 📘 Expand interface
* 📘 Relocate types
* 📘 Exempt cred entity with service-injected fields
* 📘 Adjust interface
* ♻️ Add permissions only in `GET /credentials`
* 🧪 Add expectations for `ownedBy`
* 🧪 Add sharing details test
* 🧪 Make `ownedBy` checks more granular
* 📘 Adjust interface
* 🚚 Rename cred getter
* ♻️ Refactor cred getter
* 🧪 Expand tests
* ♻️ Refactor to use guard
* 👕 Remove unneeded lint exception
* 🔥 Remove unneeded relation
* 🚚 Move relation to `GET /credentials/:id`
* 📘 Consolidate typings
* 🎨 Add multiline for readability
* 🔥 Remove unneeded type
* ✏️ Clarity comment
* ✏️ Make comments consistent
* 👕 Add exception to fix build
* 👕 Add more lint exceptions to fix build
* 🐛 Check for non-owner
* 📘 Improve typings
* 🧪 Temporarily skip tests
* 🔥 Remove `@ts-ignore`
* 👕 Move lint exceptions
* ♻️ Refactor cred service and controller
* ⚡ Simplify check
* ✏️ adjust naming to experimental
* ⚡️ add credentialsSharing flag to settings
* 🛠 add helper to check if UM is also enabled as dependency for CredentialsSharing
* 👕 fix lint error
* 🐘 change name of credential role
* 🚧 WIP batch sharing
* 🚧 WIP use put for sharing
* ✅ add tests for batch sharing, 🛠 implement review suggestions
* ✅ expand credential sharing tests for User Management (#3931)
* 🧪 Expand cred sharing tests
* ⚡ Add recently added flags
* ✅ fix and adjust tests for /credentials
Co-authored-by: Ben Hesseldieck <b.hesseldieck@gmail.com>
* ✨ User management v2 Front End (#3795)
* feat: Added responsive generic page view layout.
* feat: Added empty state.
* feat: Added credentials view empty state.
* test: Added unit tests for N8nActionBox
* feat: Added credentials list initial design.
* feat: Added credential actions. Started working on filters.
* feat: Updated InfoTip markup, added tests and changed stories to typescript.
* feat: Added credentials filtering by type. Added support for apply/reset filters.
* feat: Added credential sharing user select and user list. Added paywall component.
* feat: Updated credentials view permissions.
* feat: Added support for temporary sharing config for unsaved credentials.
* test: Fixed broken snapshots.
* feat: Added overflow styles to page-view-layout list.
* feat: Handled sharee specific views.
* feat: Integration between FE and BE to support real-world credential sharing scenario.
* feat: Added front end permissions table.
* feat: Refactored credential sharing flow. Updated design elements.
* feat: Added margin and padding auto spacer utilities.
* feat: Rehauled permissions to support instanceOwner role and action inheritance.
* feat: Updated credentials view to apply filters automatically.
* feat: Removed apply filters button and added active button state.
* test: Updated component snapshots.
* refactor: Renamed ResourceSharee to ResourceReader.
* feat: Credential sharing error handling, permissions improvement.
* feat: Updated permissions and error handling.
* chore: Removed console.log.
* 🛠 refactor enabling of credentialsSharing
* feat: Removed owner menu selector from credentials when sharing is disabled.
* refactor: Moved EE features into ee store module file.
* 🛠 add sharing info to GET credentials/:id
* fix: Fixed initial credential data loading for sharing.
* chore: Removed console.log.
* 🐛 owner can fetch any credential
* 🛠 refactor users test
* 👕 fix build type issue
* fix: Removed owner tag when credential sharing is disabled. Fixed small reactivity issue.
* chore: Removed console.log.
* 🚧 separate fetching credentials between EE and open
* fix: Fixed empty dropdown in users list.
* fix: Fixed error message and initialization when credential gets unshared.
* ✅ add tests for fetching single credential
* Revert decorators based controllers
* ⚡️ adjust credentials test route to also allow testing for sharees (#3999)
* ⚡️ pull data if user is sharee
* fix: Removed sharedWith and ownedBy from credentialData on testing credentials.
Co-authored-by: Alex Grozav <alex@grozav.com>
* 📈 add BE analytics
* 💪 improve credential test
* ⚡️ adjust tracking properties
* ⚡️ removed roles from tracking
* 🐛 fix build by removing imports
* 🐛 fix missed merge conflict
* feat: User management P2 Front End bug bash and improvements (#4014)
* fix: Fixed type select size after reopening dropdown.
* fix: Fixed template cards.
* fix: Fixed card content size and copy input.
* fix: Fixed horizontal overflow.
* fix: Hiding el-tags scrollbar in select.
* fix: Added fallback credential icon. Added oAuth credential owner check.
* feat: Added disabled state to user select.
* feat: Added fallback scenario for non-existent credential types.
* feat: Adjusted credentials empty state to show that there are shared credentials.
* fix: Fixed time title.
* feat: Added actionable empty state when shared credentials are present.
* fix: Made action box x padding smaller
* feat: Repositioned owner tag for credential card.
* feat: Updated message box styling to use n8n css variables.
* feat: Added confirmation for deleting sharee.
* fix: Fixed deleted credential types. Fixed select in dropdown bug.
* fix: Various code improvements. Addressed PR review comments.
* fix: Fixed credential deletion errors.
* fix: Various code quality improvements.
* feat: N8N-4531 update cloud coming soon features (#4025)
* feat: Showing different upcoming feature messages and format for cloud.
* fix: Changed url format.
* fix: Updated how cloud deployment is determined.
* feat: N8N-4527 implementing credential sharing FE telemetry (#4023)
* feat: Added credential sharing telemetry.
* chore: Renamed computed function for consistency.
* refactor: Simplified subview telemetry sending.
* fix: Changed to callDebounced() helper.
* 📧 update email text
* fix: Adjusted feature coming soon margin.
* chore: Fixed type and line height for delete sharee confirmation modal.
* refactor(editor-ui): Update telemetry (#4040)
* 🔥 Remove `identify` from BE
* ⚡ Add `versionCli`
* ⚡ Add node creator ignore input
* ⚡ Move obfuscators to editor-ui
* ⚡ Refactor `ph-no-capture`
* ⚡ Pass `user_id` to manual exec props
* 🚚 Relocate class in `SettingsApiView`
* ⚡ Add `userId` to BE PH `identify` call
* ⏪ Revert "⚡ Add `userId` to BE PH `identify` call"
This reverts commit 895aaa45e5.
* Revert "⏪ Revert "⚡ Add `userId` to BE PH `identify` call""
This reverts commit b86a098c20.
* 🐛 Fix `Promise` handling in `track()` call
* ⏪ Restore `Db.collections` call
* ⚡ Set up PH payload to mirror RS
* 🔥 Remove excess `userId`
* 📘 Remove `userId` from interface
* 🔥 Remove unused ref and method
* fix: Fixed bug causing instanceOwner to become credential owner on update. (#4079)
* 🐛 fix test for credential shared with member
* 👕 fix lint issues
* delete conflicting migration. this data is already seeded in CreateUserManagement
* feat: Expand obfuscation to User Management credential sharing (#4070)
⚡ Expand obfuscation
* feat: Added credential sharing infotip for instance owner.
* bring back the migration. add a check to avoid conflicts on inserts
* fix(cli): use a non-env config flag to detect of enterprise features are enabled (#4105)
* chore: Changed ampersand to and in translation.
* refactor(telemetry): Obfuscate code and JSON editors (#4118)
⚡ Obfuscate code and JSON editors
* feat(editor): improve design and functionality of coming soon features (#4116)
* feat: Improved coming soon feature design and functionality.
* style: Removed empty line.
* chore: Removed unused translation.
* fix: fix telemetry for credential creates and updates (#4125)
fix telemetry for credential creates and updates
* feat: Display errors due to missing credentials in the correct node (#4124)
feat: Display errors due to invalid credentials in the correct node when missing permissions
* fix: remove duplicate header for coming soon features in cloud deployment
* telemetry: fix the payload for `User viewed credential tab`
* telemetry: add credential_id to 'User selected credential from node modal'
* feat: update empty states for coming soon features
* Update ActionBox.spec.ts.snap
* replace UserSharingsDetails with a subset of User properties
* rename the CreateCredentialsEditorRole to CreateCredentialsUserRole
* move IUser to the workflow package
* use IUser in the frontend as well
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
Co-authored-by: Valya <68596159+valya@users.noreply.github.com>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <netroy@users.noreply.github.com>
Co-authored-by: Alex Grozav <alex@grozav.com>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Co-authored-by: Omar Ajoue <krynble@gmail.com>
fix(typescript): Use consistent typescript configs
make all packages inherit tsconfig from the root tsconfig
skips building tests. reformat all tsconfigs with prettier.
* Improve code health
Fix TS typos in local variables
Fix CSS typos in local styles
Fix typos in comments
Fix typos in strings
* Fix order of n8n setup sections in CONTRIBUTING.md
