fix: Clear input fields data operations#11773
Conversation
feat(auth): Pluggable AuthService with abstract base class (#10702) * feat: Introduce service registration decorator and enhance ServiceManager for pluggable service discovery - Added `register_service` decorator to allow services to self-register with the ServiceManager. - Enhanced `ServiceManager` to support multiple service discovery mechanisms, including decorator-based registration, config files, and entry points. - Implemented methods for direct service class registration and plugin discovery from various sources, improving flexibility and extensibility of service management. * feat: Implement VariableService for managing environment variables - Introduced VariableService class to handle environment variables with in-memory caching. - Added methods for getting, setting, deleting, and listing variables. - Included logging for service initialization and variable operations. - Created an __init__.py file to expose VariableService in the package namespace. * feat: Enhance LocalStorageService with Service integration and async teardown - Updated LocalStorageService to inherit from both StorageService and Service for improved functionality. - Added a name attribute for service identification. - Implemented an async teardown method for future extensibility, even though no cleanup is currently needed. - Refactored the constructor to ensure proper initialization of both parent classes. * feat: Implement telemetry service with abstract base class and minimal logging functionality - Added `BaseTelemetryService` as an abstract base class defining the interface for telemetry services. - Introduced `TelemetryService`, a lightweight implementation that logs telemetry events without sending data. - Created `__init__.py` to expose the telemetry service in the package namespace. - Ensured robust async methods for logging various telemetry events and handling exceptions. * feat: Introduce BaseTracingService and implement minimal TracingService - Added `BaseTracingService` as an abstract base class defining the interface for tracing services. - Implemented `TracingService`, a lightweight version that logs trace events without external integrations. - Included async methods for starting and ending traces, tracing components, and managing logs and outputs. - Enhanced documentation for clarity on method usage and parameters. * feat: Add unit tests for service registration decorators - Introduced a new test suite for validating the functionality of the @register_service decorator. - Implemented tests for various service types including LocalStorageService, TelemetryService, and TracingService. - Verified behavior for service registration with and without overrides, ensuring correct service management. - Included tests for custom service implementations and preservation of class functionality. - Enhanced overall test coverage for the service registration mechanism. * feat: Add comprehensive unit and integration tests for ServiceManager - Introduced a suite of unit tests covering edge cases for service registration, lifecycle management, and dependency resolution. - Implemented integration tests to validate service loading from configuration files and environment variables. - Enhanced test coverage for various service types including LocalStorageService, TelemetryService, and VariableService. - Verified behavior for service registration with and without overrides, ensuring correct service management. - Ensured robust handling of error conditions and edge cases in service creation and configuration parsing. * feat: Add unit and integration tests for minimal service implementations - Introduced comprehensive unit tests for LocalStorageService, TelemetryService, TracingService, and VariableService. - Implemented integration tests to validate the interaction between minimal services. - Ensured robust coverage for file operations, service readiness, and exception handling. - Enhanced documentation within tests for clarity on functionality and expected behavior. * docs: Add detailed documentation for pluggable services architecture and usage * feat: Add example configuration file for Langflow services * docs: Update PLUGGABLE_SERVICES.md to enhance architecture benefits section - Revised the documentation to highlight the advantages of the pluggable service system. - Replaced the migration guide with a detailed overview of features such as automatic discovery, lazy instantiation, dependency injection, and lifecycle management. - Clarified examples of service registration and improved overall documentation for better understanding. * [autofix.ci] apply automated fixes * test(services): improve variable service teardown test with public API assertions * docs(pluggable-service-layer): add docstrings for service manager and implementations * fix: remove duplicate teardown method from LocalStorageService During rebase, the teardown method was added in two locations (lines 57 and 220). Removed the duplicate at line 57, keeping the one at the end of the class (line 220) which is the more appropriate location for cleanup methods. * fix(tests): update service tests for LocalStorageService constructor changes - Add MockSessionService fixtures to test files that use ServiceManager - Update LocalStorageService test instantiation to use mock session and settings services - Fix service count assertions to account for MockSessionService in fixtures - Remove duplicate class-level clean_manager fixtures in test_edge_cases.py These changes fix test failures caused by LocalStorageService requiring session_service and settings_service parameters instead of just data_dir. * fix(services): Harden service lifecycle methods - Fixed Diamond Inheritance in LocalStorageService - Added Circular Dependency Detection in _create_service_from_class - Fixed StorageService.teardown to Have Default Implementation * docs: Update discovery order for pluggable services * fix(lfx): replace aiofile with aiofiles for CI compatibility - The aiofile library uses native async I/O (libaio) which fails with EAGAIN (SystemError: 11, 'Resource temporarily unavailable') in containerized environments like GitHub Actions runners. - Switch to aiofiles which uses thread pool executors, providing reliable async file I/O across all environments including containers. * [autofix.ci] apply automated fixes * fix(lfx): prevent race condition in plugin discovery The discover_plugins() method had a TOCTOU (time-of-check to time-of-use) race condition. Since get() uses a keyed lock (per service name), multiple threads requesting different services could concurrently see _plugins_discovered=False and trigger duplicate plugin discovery. Wrap discover_plugins() with self._lock to ensure thread-safe access to the _plugins_discovered flag and prevent concurrent discovery execution. * [autofix.ci] apply automated fixes * feat: Introduce service registration decorator and enhance ServiceManager for pluggable service discovery - Added `register_service` decorator to allow services to self-register with the ServiceManager. - Enhanced `ServiceManager` to support multiple service discovery mechanisms, including decorator-based registration, config files, and entry points. - Implemented methods for direct service class registration and plugin discovery from various sources, improving flexibility and extensibility of service management. * feat: Enhance LocalStorageService with Service integration and async teardown - Updated LocalStorageService to inherit from both StorageService and Service for improved functionality. - Added a name attribute for service identification. - Implemented an async teardown method for future extensibility, even though no cleanup is currently needed. - Refactored the constructor to ensure proper initialization of both parent classes. * docs(pluggable-service-layer): add docstrings for service manager and implementations * feat(auth): implement abstract base class for authentication services and add auth service retrieval function * refactor(auth): move authentication logic from utils to AuthService Consolidate all authentication methods into the AuthService class to enable pluggable authentication implementations. The utils module now contains thin wrappers that delegate to the registered auth service. This allows alternative auth implementations (e.g., OIDC) to be registered via the pluggable services system while maintaining backward compatibility with existing code that imports from utils. Changes: - Move all auth logic (token creation, user validation, API key security, password hashing, encryption) to AuthService - Refactor utils.py to delegate to get_auth_service() - Update function signatures to remove settings_service parameter (now obtained from the service internally) * refactor(auth): update authentication methods and remove settings_service parameter - Changed function to retrieve current user from access token instead of JWT. - Updated AuthServiceFactory to specify SettingsService type in create method. - Removed settings_service dependency from encryption and decryption functions, simplifying the code. This refactor enhances the clarity and maintainability of the authentication logic. * test(auth): add unit tests for AuthService and pluggable authentication - Introduced comprehensive unit tests for AuthService, covering token creation, user validation, and authentication methods. - Added tests for pluggable authentication, ensuring correct delegation to registered services. - Enhanced test coverage for user authentication scenarios, including active/inactive user checks and token validation. These additions improve the reliability and maintainability of the authentication system. * fix(tests): update test cases to use AuthService and correct user retrieval method - Replaced the mock for retrieving the current user from JWT to access token in the TestSuperuserCommand. - Refactored unit tests for MCP encryption to utilize AuthService instead of a mock settings service, enhancing test reliability. - Updated patch decorators in tests to reflect the new method of obtaining the AuthService, ensuring consistency across test cases. These changes improve the accuracy and maintainability of the authentication tests. * docs(pluggable-services): add auth_service to ServiceType enum documentation * fix(auth): Add missing type hints and abstract methods to AuthServiceBase (#10710) * [autofix.ci] apply automated fixes * fix(auth): refactor api_key_security method to accept optional database session and improve error handling * feat(auth): enhance AuthServiceBase with detailed design principles and JIT provisioning methods * fix(auth): remove settings_service from encrypt/decrypt_api_key calls After the pluggable auth refactor, encrypt_api_key and decrypt_api_key no longer take a settings_service argument - they get it internally. - Update check_key import path in __main__.py (moved to crud module) - Remove settings_service argument from calls in: - api/v1/api_key.py - api/v1/store.py - services/variable/service.py - services/variable/kubernetes.py - Fix auth service to use session_scope() instead of non-existent get_db_service().with_session() * fix(auth): resolve type errors and duplicate definitions in pluggable auth branch - Add missing imports in auth/utils.py (Final, HTTPException, status, logger, SettingsService) that prevented application startup - Remove duplicate NoServiceRegisteredError class in lfx/services/manager.py - Remove duplicate teardown method in lfx/services/storage/local.py - Fix invalid settings_service parameter in encrypt_api_key calls in variable/service.py and variable/kubernetes.py - Add proper type guards for check_key calls to satisfy mypy - Add null checks for password fields in users.py endpoints * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * [autofix.ci] apply automated fixes (attempt 3/3) * [autofix.ci] apply automated fixes * replace jose with pyjwt * [autofix.ci] apply automated fixes * starter projects * fix BE mcp tests * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * remive legacy usage of session * fix user tests * [autofix.ci] apply automated fixes * fix lfx tests * starter project update * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * fix mypy errors * fix mypy errors on tests * fix tests for decrypt_api_key * resolve conflicts in auth utils * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * Add pluggable authentication factory with provider enum * Add SSO feature flags to AuthSettings * Add SSO fields to User model * Add SSO configuration loader with YAML support * Add unit tests for SSO configuration loader * Add SSO configuration database model and CRUD operations * Add CRUD operations for SSO configuration management * Add SSO configuration service supporting both file and database configs * Add example SSO configuration file with W3ID and other providers * Implement OIDC authentication service with discovery and JIT provisioning * Update AuthServiceFactory to instantiate OIDC service when SSO enabled * Improve JWT token validation and API key decryption error handling * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes * fix: resolve ruff linting errors in auth services and add sso-config.yaml to gitignore * [autofix.ci] apply automated fixes * fix: use correct function name get_current_user_from_access_token in login endpoint * fix: remove incorrect settings_service parameter from decrypt_api_key call * fix: correct encryption logic to properly detect plaintext vs encrypted values * [autofix.ci] apply automated fixes * fix tests * [autofix.ci] apply automated fixes * fix mypy errors * fix tests * [autofix.ci] apply automated fixes * fix ruff errors * fix tests in service * [autofix.ci] apply automated fixes * fix test security cors * [autofix.ci] apply automated fixes * fix webhook issues * modify component index * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * [autofix.ci] apply automated fixes (attempt 3/3) * fix webhook tests * [autofix.ci] apply automated fixes * build component index * remove SSO functionality * [autofix.ci] apply automated fixes * fix variable creation * [autofix.ci] apply automated fixes * refactor: move MCPServerConfig schema to a separate file and update model_dump usage * refactor: streamline AuthServiceFactory to use service_class for instance creation * handle access token type * [autofix.ci] apply automated fixes * remove SSO fields from user model * [autofix.ci] apply automated fixes * replace is_encrypted back * fix mypy errors * remove sso config example * feat: Refactor framework agnostic auth service (#11565) * modify auth service layer * [autofix.ci] apply automated fixes * fix ruff errorrs * [autofix.ci] apply automated fixes * Update src/backend/base/langflow/services/deps.py * address review comments * [autofix.ci] apply automated fixes * fix ruff errors * remove cache --------- * move base to lfx * [autofix.ci] apply automated fixes * resolve review comments * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * add auth protocol * [autofix.ci] apply automated fixes * revert models.py execption handling * revert wrappers to ensure backwards compatibility * fix http error code * fix FE tests * fix test_variables.py * [autofix.ci] apply automated fixes * fix ruff errors * fix tests * add wrappers for create token methods * fix ruff errors * [autofix.ci] apply automated fixes * update error message * modify status code for inactive user * fix ruff errors * fix patch for webhook tests * fix error message when getting active users --------- Co-authored-by: Gabriel Luiz Freitas Almeida <gabriel@logspace.ai> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Mike Pawlowski <mike.pawlowski@datastax.com> Co-authored-by: Mike Pawlowski <mpawlow@ca.ibm.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: ogabrielluiz <24829397+ogabrielluiz@users.noreply.github.com> Co-authored-by: Deon Sanchez <69873175+deon-sanchez@users.noreply.github.com> Co-authored-by: codeflash-ai[bot] <148906541+codeflash-ai[bot]@users.noreply.github.com>
* revert textarea to old classes * fixed text-area-wrapper to handle initial height when value is calculated * fixed playground padding * fixed no input text size * [autofix.ci] apply automated fixes * fixed flaky test --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
* Create guardrails.py * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * Update guardrails.py * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * tests: add unit tests for GuardrailsComponent functionality * [autofix.ci] apply automated fixes * fix: resolve linting errors in GuardrailsComponent and tests - Fix line length issues (E501) by breaking long strings - Fix docstring formatting (D205, D415) in _check_guardrail - Use ternary operator for response content extraction (SIM108) - Replace magic value with named constant (PLR2004) - Move return to else block per try/except best practices (TRY300) - Catch specific exceptions instead of blind Exception (BLE001) - Use list comprehension for checks_to_run (PERF401) - Mark unused variables with underscore prefix (RUF059, F841) - Add noqa comment for intentionally unused mock argument (ARG002) * [autofix.ci] apply automated fixes * refactor: address pr comments * [autofix.ci] apply automated fixes (attempt 2/3) * [autofix.ci] apply automated fixes * feat: enhance heuristic detection with configurable threshold and scoring system * refactor: simplify heuristic test assertions by removing unused variable * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * feat: enhance guardrail validation logic and input handling * refactor: streamline import statements and clean up whitespace in guardrails component * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * Fix: update empty input handling tests to raise ValueError and refactor related assertions * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes * feat: add Guardrails component with unit tests Add LLM-based guardrails component for detecting PII, tokens/passwords, jailbreak attempts, and custom guardrail rules, along with comprehensive unit tests. * [autofix.ci] apply automated fixes * fix: try removing logs * [autofix.ci] apply automated fixes --------- Co-authored-by: Lucas Democh <ldgoularte@gmail.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
* Implemented dismiss file functionality on input file component * fixed hover behavior * added test for removing file from input * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
* fixed react flow utils to clean advanced edges * Make connected handles not be able to be hidden * Added test for hiding connected handles * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
fix tooltip showing up when closing select
* fix(frontend): prevent crash when renaming empty sessions * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
…#11722) The regex in langflow_pre_release_tag.py expected a dot before `rc` (e.g. `1.8.0.rc0`), but PyPI returns PEP 440-normalized versions without the dot (e.g. `1.8.0rc0`). This caused the script to recompute the same version instead of incrementing, and `uv publish` silently skipped the duplicate upload. Update the regex to accept both formats with `\.?rc`.
…11725) * fix: align chat history with input field in fullscreen playground * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
* fix singleton webhook on flow * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
…Variable button (#11723) * fix: generate unique variable names in Prompt Template Add Variable button Previously, clicking the Add Variable button always inserted {variable_name}, causing duplicate text without creating new input fields. Now the button generates incremental names (variable_name, variable_name_1, variable_name_2) by checking existing variables in the template. * refactor: extract generateUniqueVariableName and import in tests Extract the variable name generation logic into an exported function so tests can import and validate the actual production code instead of testing a duplicated copy of the logic.
…11709) add edge between components Co-authored-by: Olayinka Adelakun <olayinkaadelakun@Olayinkas-MacBook-Pro.local>
) * Update state when exiting modal on accordion prompt component * Added isDoubleBrackets and show correct modal and use correct brackets when mustache is enabled * [autofix.ci] apply automated fixes * added test to see if state is synchronized and mustache is enabled * [autofix.ci] apply automated fixes * updated mustache id and removed extra prompt call * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Carlos Coelho <80289056+carlosrcoelho@users.noreply.github.com>
…es (#11720) * fix(frontend): add Safari-specific padding for playground chat messages * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
* Correctly parse dicts from tweaks * Add test * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
fix: sessions overflow issue
…tion (#11751) * merge fix * code improvements * [autofix.ci] apply automated fixes * add stop button and fix scroll on message * [autofix.ci] apply automated fixes * add new message content for sharable pg * fix tests until shard 43 * [autofix.ci] apply automated fixes * fix(frontend): clean up MemoizedSidebarTrigger imports and transition classes Sort imports, add type modifier to AllNodeType import, and split long transition class string for readability. * fix tests * [autofix.ci] apply automated fixes * fix mr test * fix jest tests * fix sidebar jest tes * [autofix.ci] apply automated fixes * fix sharable playground * [autofix.ci] apply automated fixes * remove rename from sharable pg * [autofix.ci] apply automated fixes * add new message content for sharable pg * fix: synchronize prompt state, add new mustache prompt component (#11702) * Update state when exiting modal on accordion prompt component * Added isDoubleBrackets and show correct modal and use correct brackets when mustache is enabled * [autofix.ci] apply automated fixes * added test to see if state is synchronized and mustache is enabled * [autofix.ci] apply automated fixes * updated mustache id and removed extra prompt call * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Carlos Coelho <80289056+carlosrcoelho@users.noreply.github.com> * fix(frontend): add Safari-specific padding for playground chat messages (#11720) * fix(frontend): add Safari-specific padding for playground chat messages * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix: correctly pass headers in mcp stdio connections (#11746) * fix sharable playground * [autofix.ci] apply automated fixes * remove rename from sharable pg * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes * fix sharable playground * fix mcp server to use shell lexer * [autofix.ci] apply automated fixes * fix tests * fix outaded component tests --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Viktor Avelino <viktor.avelino@gmail.com> Co-authored-by: Lucas Oliveira <62335616+lucaseduoli@users.noreply.github.com> Co-authored-by: Carlos Coelho <80289056+carlosrcoelho@users.noreply.github.com> Co-authored-by: keval shah <kevalvirat@gmail.com> Co-authored-by: Jordan Frazier <122494242+jordanrfrazier@users.noreply.github.com>
* fix: correct field_order in all starter project JSON templates The field_order arrays in starter project nodes were out of sync with the actual input definitions in the Python component source files, causing parameters to display in the wrong order in the UI. Fixed 136 nodes across 32 starter project files including Chat Input, Chat Output, Language Model, Agent, Prompt Template, Text Input, Tavily AI Search, Read File, Embedding Model, and others. * test: add field_order validation test for starter projects Verifies that field_order arrays in starter project JSONs match the actual component input order by importing each component and comparing the relative ordering of fields. * fix mcp server to use shell lexer * [autofix.ci] apply automated fixes * fix: enforce full field_order in starter projects and add node overlap test Update all starter project JSONs to include the complete component field_order instead of a subset, preventing layout inconsistency between template and sidebar. Strengthen the field_order test to require an exact match and add a new test that verifies no two generic nodes overlap on the canvas. --------- Co-authored-by: cristhianzl <cristhian.lousa@gmail.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
* Fix dict handling of different formats * [autofix.ci] apply automated fixes * [autofix.ci] apply automated fixes (attempt 2/3) * [autofix.ci] apply automated fixes (attempt 3/3) * cmp index * [autofix.ci] apply automated fixes --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
improve styling of templete input Co-authored-by: Olayinka Adelakun <olayinkaadelakun@Olayinkas-MacBook-Pro.local>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* move chat input arround for travel json starter template * improve the layout of the component * fix layout --------- Co-authored-by: Olayinka Adelakun <olayinkaadelakun@Olayinkas-MacBook-Pro.local>
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Use the checkbox below for a quick retry:
WalkthroughThis PR adds explicit UI state management for operation-specific fields in the DataOperationsComponent. When the operations array is cleared, operation-related fields are now hidden and reset to default values. Changes span backend logic, frontend UI updates, test coverage, and asset metadata. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error, 3 warnings, 1 inconclusive)
✅ Passed checks (2 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
❌ Your patch status has failed because the patch coverage (0.00%) is below the target coverage (40.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #11773 +/- ##
==========================================
- Coverage 35.31% 35.31% -0.01%
==========================================
Files 1525 1525
Lines 73292 73299 +7
Branches 11021 11025 +4
==========================================
+ Hits 25884 25885 +1
- Misses 45995 46000 +5
- Partials 1413 1414 +1
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
viktoravelino
left a comment
There was a problem hiding this comment.
fe code looks good to me
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (5)
src/lfx/src/lfx/components/processing/data_operations.py (2)
498-503: Redundant inner check: everyALL_OPERATION_FIELDSentry exists inOPERATION_FIELD_DEFAULTS.The
if field in self.OPERATION_FIELD_DEFAULTSguard on line 502 is alwaysTruebecause both lists are defined with identical keys. Simplify to a direct access or use.get()for defensive coding:Suggested simplification
for field in self.ALL_OPERATION_FIELDS: if field in build_config: build_config[field]["show"] = False - if field in self.OPERATION_FIELD_DEFAULTS: - build_config[field]["value"] = self.OPERATION_FIELD_DEFAULTS[field] + build_config[field]["value"] = deepcopy(self.OPERATION_FIELD_DEFAULTS[field])🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/lfx/src/lfx/components/processing/data_operations.py` around lines 498 - 503, The inner conditional checking membership in OPERATION_FIELD_DEFAULTS is redundant because ALL_OPERATION_FIELDS entries are guaranteed to exist there; simplify the loop in the method that iterates ALL_OPERATION_FIELDS by directly assigning build_config[field]["value"] = self.OPERATION_FIELD_DEFAULTS[field] (or use self.OPERATION_FIELD_DEFAULTS.get(field) for defensive style) after setting build_config[field]["show"] = False; this change affects the block referencing ALL_OPERATION_FIELDS, OPERATION_FIELD_DEFAULTS, and build_config in data_operations.py.
248-261: DRY:OPERATION_FIELD_DEFAULTSduplicates values from input declarations.The defaults for
operator,append_update_data,rename_keys_input, etc. are already declared in theinputslist (lines 135–247). Maintaining them in two places invites divergence. Consider deriving defaults programmatically from the input definitions:OPERATION_FIELD_DEFAULTS = { inp.name: inp.value for inp in inputs if inp.name in ALL_OPERATION_FIELDS }This was also raised in the PR review comments by Cristhianzl.
Also applies to: 84-96
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/lfx/src/lfx/components/processing/data_operations.py` around lines 248 - 261, OPERATION_FIELD_DEFAULTS currently duplicates default values already declared on the Input definitions; replace the hard-coded dict with a derived one by iterating over the inputs list and selecting entries whose names appear in ALL_OPERATION_FIELDS (e.g., OPERATION_FIELD_DEFAULTS = {inp.name: inp.value for inp in inputs if inp.name in ALL_OPERATION_FIELDS}), and apply the same replacement for the earlier duplicated block around the symbols referenced at lines 84–96; ensure the derived defaults preserve the original value objects (not copies) and keep the same key set as the original constant.src/frontend/src/CustomNodes/hooks/use-handle-new-value.ts (1)
16-28: Cross-stack duplication: field list must be kept in sync manually.
DATA_OPERATIONS_OPERATION_FIELDSduplicatesALL_OPERATION_FIELDSfromdata_operations.py. The sync comment helps, but any backend field addition/removal will silently desync the frontend. Consider deriving this list from the node template at runtime (e.g., fromnode.templateor an API-exposed field list) to eliminate the manual coupling.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/frontend/src/CustomNodes/hooks/use-handle-new-value.ts` around lines 16 - 28, DATA_OPERATIONS_OPERATION_FIELDS in use-handle-new-value.ts duplicates ALL_OPERATION_FIELDS from data_operations.py and must be kept manually in sync; replace the hardcoded array by deriving the field list at runtime from the node template or an API to avoid drift — e.g., update the logic in useHandleNewValue (or the module exporting DATA_OPERATIONS_OPERATION_FIELDS) to read node.template (or call the backend endpoint that returns operation field names) and build the equivalent array dynamically, falling back to the current static list only if the template/API is unavailable.src/lfx/src/lfx/_assets/component_index.json (2)
95583-95583:OPERATION_FIELD_DEFAULTSduplicates the default values already declared in theinputslist.For example,
"operator": "equals"appears in bothOPERATION_FIELD_DEFAULTSand theDropdownInput(name="operator", ..., value="equals")definition. If a default ever changes in one place but not the other, the reset behavior will silently diverge. The reviewer already suggested deriving defaults programmatically from theinputsdefinitions. Consider adopting that approach to keep a single source of truth.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/lfx/src/lfx/_assets/component_index.json` at line 95583, OPERATION_FIELD_DEFAULTS duplicates defaults defined in the inputs list (e.g., "operator" default "equals"); replace the hard-coded OPERATION_FIELD_DEFAULTS with a programmatic generator that iterates over the inputs list and builds the defaults map from each input's declared value (and appropriate empty defaults for is_list inputs), ensuring keys from ALL_OPERATION_FIELDS are included; update the class to compute this generated defaults (e.g., at class definition time or in a small helper like build_operation_field_defaults()) and keep update_build_config using that generated OPERATION_FIELD_DEFAULTS so there is a single source of truth for default values.
95583-95583: Redundantif field in self.OPERATION_FIELD_DEFAULTScheck insideupdate_build_config.Every entry in
ALL_OPERATION_FIELDShas a corresponding key inOPERATION_FIELD_DEFAULTS, so the inner guard is alwaysTrue. This was also flagged by the reviewer. Simplify by removing the redundant check or, for extra safety, use.get().Proposed simplification
In the
update_build_configmethod:for field in self.ALL_OPERATION_FIELDS: if field in build_config: build_config[field]["show"] = False - if field in self.OPERATION_FIELD_DEFAULTS: - build_config[field]["value"] = self.OPERATION_FIELD_DEFAULTS[field] + build_config[field]["value"] = self.OPERATION_FIELD_DEFAULTS[field]🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/lfx/src/lfx/_assets/component_index.json` at line 95583, The loop in update_build_config iterates ALL_OPERATION_FIELDS and then redundantly checks if field in OPERATION_FIELD_DEFAULTS before resetting values; remove that inner guard and directly reset show and set the default using OPERATION_FIELD_DEFAULTS.get(field) (or assume presence and index OPERATION_FIELD_DEFAULTS[field]) so each field is hidden and its value reset consistently; update_build_config, ALL_OPERATION_FIELDS and OPERATION_FIELD_DEFAULTS are the symbols to change.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In
`@src/backend/tests/unit/components/processing/test_data_operations_component.py`:
- Around line 195-223: The test for DataOperationsComponent.update_build_config
is incomplete: extend the mock build_config to include all operation-specific
keys (append_update_data, remove_keys_input, rename_keys_input,
mapped_json_display, selected_key, query, etc.) so their show flags are
exercised, and add assertions that values are reset to OPERATION_FIELD_DEFAULTS
(e.g., assert result["operator"]["value"] equals the expected default and
similarly for other fields) after calling component.update_build_config;
reference the DataOperationsComponent class and its update_build_config method
and use the same field names (filter_key, operator, filter_values,
select_keys_input, append_update_data, remove_keys_input, rename_keys_input,
mapped_json_display, selected_key, query) to add both show=false and
value==OPERATION_FIELD_DEFAULTS checks.
In `@src/lfx/src/lfx/_assets/component_index.json`:
- Line 95583: The as_data method assumes every item in self.operations is a dict
with a "name" key which can raise KeyError/TypeError; change the
selected_actions extraction in as_data to the same defensive pattern used in
update_build_config (filtering with isinstance(action, dict) and "name" in
action) when building selected_actions from self.operations, ensuring you handle
empty/malformed entries gracefully before looking up action_map and invoking
handlers.
In `@src/lfx/src/lfx/components/processing/data_operations.py`:
- Around line 248-261: OPERATION_FIELD_DEFAULTS currently contains mutable
shared instances and the assignment build_config[field]["value"] =
self.OPERATION_FIELD_DEFAULTS[field] binds those shared objects; change the
assignment to provide a fresh copy (e.g., use
copy.deepcopy(self.OPERATION_FIELD_DEFAULTS[field]) or replace
OPERATION_FIELD_DEFAULTS with a function like get_operation_field_defaults()
that returns new dict/list instances) so each build_config gets its own
independent mutable objects and class-level defaults are not mutated by
consumers.
---
Nitpick comments:
In `@src/frontend/src/CustomNodes/hooks/use-handle-new-value.ts`:
- Around line 16-28: DATA_OPERATIONS_OPERATION_FIELDS in use-handle-new-value.ts
duplicates ALL_OPERATION_FIELDS from data_operations.py and must be kept
manually in sync; replace the hardcoded array by deriving the field list at
runtime from the node template or an API to avoid drift — e.g., update the logic
in useHandleNewValue (or the module exporting DATA_OPERATIONS_OPERATION_FIELDS)
to read node.template (or call the backend endpoint that returns operation field
names) and build the equivalent array dynamically, falling back to the current
static list only if the template/API is unavailable.
In `@src/lfx/src/lfx/_assets/component_index.json`:
- Line 95583: OPERATION_FIELD_DEFAULTS duplicates defaults defined in the inputs
list (e.g., "operator" default "equals"); replace the hard-coded
OPERATION_FIELD_DEFAULTS with a programmatic generator that iterates over the
inputs list and builds the defaults map from each input's declared value (and
appropriate empty defaults for is_list inputs), ensuring keys from
ALL_OPERATION_FIELDS are included; update the class to compute this generated
defaults (e.g., at class definition time or in a small helper like
build_operation_field_defaults()) and keep update_build_config using that
generated OPERATION_FIELD_DEFAULTS so there is a single source of truth for
default values.
- Line 95583: The loop in update_build_config iterates ALL_OPERATION_FIELDS and
then redundantly checks if field in OPERATION_FIELD_DEFAULTS before resetting
values; remove that inner guard and directly reset show and set the default
using OPERATION_FIELD_DEFAULTS.get(field) (or assume presence and index
OPERATION_FIELD_DEFAULTS[field]) so each field is hidden and its value reset
consistently; update_build_config, ALL_OPERATION_FIELDS and
OPERATION_FIELD_DEFAULTS are the symbols to change.
In `@src/lfx/src/lfx/components/processing/data_operations.py`:
- Around line 498-503: The inner conditional checking membership in
OPERATION_FIELD_DEFAULTS is redundant because ALL_OPERATION_FIELDS entries are
guaranteed to exist there; simplify the loop in the method that iterates
ALL_OPERATION_FIELDS by directly assigning build_config[field]["value"] =
self.OPERATION_FIELD_DEFAULTS[field] (or use
self.OPERATION_FIELD_DEFAULTS.get(field) for defensive style) after setting
build_config[field]["show"] = False; this change affects the block referencing
ALL_OPERATION_FIELDS, OPERATION_FIELD_DEFAULTS, and build_config in
data_operations.py.
- Around line 248-261: OPERATION_FIELD_DEFAULTS currently duplicates default
values already declared on the Input definitions; replace the hard-coded dict
with a derived one by iterating over the inputs list and selecting entries whose
names appear in ALL_OPERATION_FIELDS (e.g., OPERATION_FIELD_DEFAULTS =
{inp.name: inp.value for inp in inputs if inp.name in ALL_OPERATION_FIELDS}),
and apply the same replacement for the earlier duplicated block around the
symbols referenced at lines 84–96; ensure the derived defaults preserve the
original value objects (not copies) and keep the same key set as the original
constant.
| def test_update_build_config_clears_input_fields_when_operation_removed(self): | ||
| """Test that removing the selected operation hides all operation-specific input fields.""" | ||
| from lfx.schema.dotdict import dotdict | ||
|
|
||
| component = DataOperationsComponent( | ||
| data=Data(data={"key1": "value1"}), | ||
| operations=[], | ||
| ) | ||
| # Simulate build_config after "Filter Values" was selected (operation-specific fields visible) | ||
| build_config = dotdict( | ||
| { | ||
| "operations": {"value": [], "show": True}, | ||
| "data": {"value": None, "show": True}, | ||
| "filter_key": {"value": [], "show": True}, | ||
| "operator": {"value": "equals", "show": True}, | ||
| "filter_values": {"value": {}, "show": True}, | ||
| "select_keys_input": {"value": [], "show": False}, | ||
| } | ||
| ) | ||
| result = component.update_build_config(build_config, [], "operations") | ||
|
|
||
| # All operation-specific fields should be hidden when no operation is selected | ||
| assert result["filter_key"]["show"] is False | ||
| assert result["operator"]["show"] is False | ||
| assert result["filter_values"]["show"] is False | ||
| assert result["select_keys_input"]["show"] is False | ||
| # Default fields (operations, data) should remain visible | ||
| assert result["operations"]["show"] is True | ||
| assert result["data"]["show"] is True |
There was a problem hiding this comment.
🛠️ Refactor suggestion | 🟠 Major
Test coverage gaps: missing fields and value-reset assertions.
Two gaps worth addressing:
-
Only 4 of 10 operation fields are included in the mock
build_config. Fields likeappend_update_data,remove_keys_input,rename_keys_input,mapped_json_display,selected_key, andqueryare absent, so their hide behavior goes untested. -
Value reset is not asserted. The
update_build_configmethod also resets field values toOPERATION_FIELD_DEFAULTS, but this test only checksshow. Adding assertions likeassert result["operator"]["value"] == "equals"would confirm the reset path.
Suggested additions
build_config = dotdict(
{
"operations": {"value": [], "show": True},
"data": {"value": None, "show": True},
"filter_key": {"value": [], "show": True},
"operator": {"value": "equals", "show": True},
"filter_values": {"value": {}, "show": True},
"select_keys_input": {"value": [], "show": False},
+ "append_update_data": {"value": {"key": "value"}, "show": True},
+ "remove_keys_input": {"value": ["something"], "show": True},
+ "rename_keys_input": {"value": {"a": "b"}, "show": True},
+ "mapped_json_display": {"value": "some json", "show": True},
+ "selected_key": {"value": ".foo", "show": True},
+ "query": {"value": ".bar", "show": True},
}
)
result = component.update_build_config(build_config, [], "operations")
# All operation-specific fields should be hidden when no operation is selected
assert result["filter_key"]["show"] is False
assert result["operator"]["show"] is False
assert result["filter_values"]["show"] is False
assert result["select_keys_input"]["show"] is False
+ assert result["append_update_data"]["show"] is False
+ assert result["remove_keys_input"]["show"] is False
+ assert result["rename_keys_input"]["show"] is False
+ assert result["mapped_json_display"]["show"] is False
+ assert result["selected_key"]["show"] is False
+ assert result["query"]["show"] is False
+
+ # Values should be reset to defaults
+ assert result["operator"]["value"] == "equals"
+ assert result["remove_keys_input"]["value"] == []
+ assert result["rename_keys_input"]["value"] == {"old_key": "new_key"}
+ assert result["query"]["value"] == ""
# Default fields (operations, data) should remain visible
assert result["operations"]["show"] is True
assert result["data"]["show"] is True🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In
`@src/backend/tests/unit/components/processing/test_data_operations_component.py`
around lines 195 - 223, The test for DataOperationsComponent.update_build_config
is incomplete: extend the mock build_config to include all operation-specific
keys (append_update_data, remove_keys_input, rename_keys_input,
mapped_json_display, selected_key, query, etc.) so their show flags are
exercised, and add assertions that values are reset to OPERATION_FIELD_DEFAULTS
(e.g., assert result["operator"]["value"] equals the expected default and
similarly for other fields) after calling component.update_build_config;
reference the DataOperationsComponent class and its update_build_config method
and use the same field names (filter_key, operator, filter_values,
select_keys_input, append_update_data, remove_keys_input, rename_keys_input,
mapped_json_display, selected_key, query) to add both show=false and
value==OPERATION_FIELD_DEFAULTS checks.
| "title_case": false, | ||
| "type": "code", | ||
| "value": "import ast\nimport json\nfrom typing import TYPE_CHECKING, Any\n\nimport jq\nfrom json_repair import repair_json\n\nfrom lfx.custom import Component\nfrom lfx.inputs import DictInput, DropdownInput, MessageTextInput, SortableListInput\nfrom lfx.io import DataInput, MultilineInput, Output\nfrom lfx.log.logger import logger\nfrom lfx.schema import Data\nfrom lfx.schema.dotdict import dotdict\nfrom lfx.utils.component_utils import set_current_fields, set_field_display\n\nif TYPE_CHECKING:\n from collections.abc import Callable\n\nACTION_CONFIG = {\n \"Select Keys\": {\"is_list\": False, \"log_msg\": \"setting filter fields\"},\n \"Literal Eval\": {\"is_list\": False, \"log_msg\": \"setting evaluate fields\"},\n \"Combine\": {\"is_list\": True, \"log_msg\": \"setting combine fields\"},\n \"Filter Values\": {\"is_list\": False, \"log_msg\": \"setting filter values fields\"},\n \"Append or Update\": {\"is_list\": False, \"log_msg\": \"setting Append or Update fields\"},\n \"Remove Keys\": {\"is_list\": False, \"log_msg\": \"setting remove keys fields\"},\n \"Rename Keys\": {\"is_list\": False, \"log_msg\": \"setting rename keys fields\"},\n \"Path Selection\": {\"is_list\": False, \"log_msg\": \"setting mapped key extractor fields\"},\n \"JQ Expression\": {\"is_list\": False, \"log_msg\": \"setting parse json fields\"},\n}\nOPERATORS = {\n \"equals\": lambda a, b: str(a) == str(b),\n \"not equals\": lambda a, b: str(a) != str(b),\n \"contains\": lambda a, b: str(b) in str(a),\n \"starts with\": lambda a, b: str(a).startswith(str(b)),\n \"ends with\": lambda a, b: str(a).endswith(str(b)),\n}\n\n\nclass DataOperationsComponent(Component):\n display_name = \"Data Operations\"\n description = \"Perform various operations on a Data object.\"\n icon = \"file-json\"\n name = \"DataOperations\"\n default_keys = [\"operations\", \"data\"]\n metadata = {\n \"keywords\": [\n \"data\",\n \"operations\",\n \"filter values\",\n \"Append or Update\",\n \"remove keys\",\n \"rename keys\",\n \"select keys\",\n \"literal eval\",\n \"combine\",\n \"filter\",\n \"append\",\n \"update\",\n \"remove\",\n \"rename\",\n \"data operations\",\n \"data manipulation\",\n \"data transformation\",\n \"data filtering\",\n \"data selection\",\n \"data combination\",\n \"Parse JSON\",\n \"JSON Query\",\n \"JQ Query\",\n ],\n }\n actions_data = {\n \"Select Keys\": [\"select_keys_input\", \"operations\"],\n \"Literal Eval\": [],\n \"Combine\": [],\n \"Filter Values\": [\"filter_values\", \"operations\", \"operator\", \"filter_key\"],\n \"Append or Update\": [\"append_update_data\", \"operations\"],\n \"Remove Keys\": [\"remove_keys_input\", \"operations\"],\n \"Rename Keys\": [\"rename_keys_input\", \"operations\"],\n \"Path Selection\": [\"mapped_json_display\", \"selected_key\", \"operations\"],\n \"JQ Expression\": [\"query\", \"operations\"],\n }\n\n @staticmethod\n def extract_all_paths(obj, path=\"\"):\n paths = []\n if isinstance(obj, dict):\n for k, v in obj.items():\n new_path = f\"{path}.{k}\" if path else f\".{k}\"\n paths.append(new_path)\n paths.extend(DataOperationsComponent.extract_all_paths(v, new_path))\n elif isinstance(obj, list) and obj:\n new_path = f\"{path}[0]\"\n paths.append(new_path)\n paths.extend(DataOperationsComponent.extract_all_paths(obj[0], new_path))\n return paths\n\n @staticmethod\n def remove_keys_recursive(obj, keys_to_remove):\n if isinstance(obj, dict):\n return {\n k: DataOperationsComponent.remove_keys_recursive(v, keys_to_remove)\n for k, v in obj.items()\n if k not in keys_to_remove\n }\n if isinstance(obj, list):\n return [DataOperationsComponent.remove_keys_recursive(item, keys_to_remove) for item in obj]\n return obj\n\n @staticmethod\n def rename_keys_recursive(obj, rename_map):\n if isinstance(obj, dict):\n return {\n rename_map.get(k, k): DataOperationsComponent.rename_keys_recursive(v, rename_map)\n for k, v in obj.items()\n }\n if isinstance(obj, list):\n return [DataOperationsComponent.rename_keys_recursive(item, rename_map) for item in obj]\n return obj\n\n inputs = [\n DataInput(name=\"data\", display_name=\"Data\", info=\"Data object to filter.\", required=True, is_list=True),\n SortableListInput(\n name=\"operations\",\n display_name=\"Operations\",\n placeholder=\"Select Operation\",\n info=\"List of operations to perform on the data.\",\n options=[\n {\"name\": \"Select Keys\", \"icon\": \"lasso-select\"},\n {\"name\": \"Literal Eval\", \"icon\": \"braces\"},\n {\"name\": \"Combine\", \"icon\": \"merge\"},\n {\"name\": \"Filter Values\", \"icon\": \"filter\"},\n {\"name\": \"Append or Update\", \"icon\": \"circle-plus\"},\n {\"name\": \"Remove Keys\", \"icon\": \"eraser\"},\n {\"name\": \"Rename Keys\", \"icon\": \"pencil-line\"},\n {\"name\": \"Path Selection\", \"icon\": \"mouse-pointer\"},\n {\"name\": \"JQ Expression\", \"icon\": \"terminal\"},\n ],\n real_time_refresh=True,\n limit=1,\n ),\n # select keys inputs\n MessageTextInput(\n name=\"select_keys_input\",\n display_name=\"Select Keys\",\n info=\"List of keys to select from the data. Only top-level keys can be selected.\",\n show=False,\n is_list=True,\n ),\n # filter values inputs\n MessageTextInput(\n name=\"filter_key\",\n display_name=\"Filter Key\",\n info=(\n \"Name of the key containing the list to filter. \"\n \"It must be a top-level key in the JSON and its value must be a list.\"\n ),\n is_list=True,\n show=False,\n ),\n DropdownInput(\n name=\"operator\",\n display_name=\"Comparison Operator\",\n options=[\"equals\", \"not equals\", \"contains\", \"starts with\", \"ends with\"],\n info=\"The operator to apply for comparing the values.\",\n value=\"equals\",\n advanced=False,\n show=False,\n ),\n DictInput(\n name=\"filter_values\",\n display_name=\"Filter Values\",\n info=\"List of values to filter by.\",\n show=False,\n is_list=True,\n ),\n # update/ Append data inputs\n DictInput(\n name=\"append_update_data\",\n display_name=\"Append or Update\",\n info=\"Data to append or update the existing data with. Only top-level keys are checked.\",\n show=False,\n value={\"key\": \"value\"},\n is_list=True,\n ),\n # remove keys inputs\n MessageTextInput(\n name=\"remove_keys_input\",\n display_name=\"Remove Keys\",\n info=\"List of keys to remove from the data.\",\n show=False,\n is_list=True,\n ),\n # rename keys inputs\n DictInput(\n name=\"rename_keys_input\",\n display_name=\"Rename Keys\",\n info=\"List of keys to rename in the data.\",\n show=False,\n is_list=True,\n value={\"old_key\": \"new_key\"},\n ),\n MultilineInput(\n name=\"mapped_json_display\",\n display_name=\"JSON to Map\",\n info=\"Paste or preview your JSON here to explore its structure and select a path for extraction.\",\n required=False,\n refresh_button=True,\n real_time_refresh=True,\n placeholder=\"Add a JSON example.\",\n show=False,\n ),\n DropdownInput(\n name=\"selected_key\", display_name=\"Select Path\", options=[], required=False, dynamic=True, show=False\n ),\n MessageTextInput(\n name=\"query\",\n display_name=\"JQ Expression\",\n info=\"JSON Query to filter the data. Used by Parse JSON operation.\",\n placeholder=\"e.g., .properties.id\",\n show=False,\n ),\n ]\n outputs = [\n Output(display_name=\"Data\", name=\"data_output\", method=\"as_data\"),\n ]\n\n # Helper methods for data operations\n def get_data_dict(self) -> dict:\n \"\"\"Extract data dictionary from Data object.\"\"\"\n data = self.data[0] if isinstance(self.data, list) and len(self.data) == 1 else self.data\n return data.model_dump()\n\n def json_query(self) -> Data:\n import json\n\n import jq\n\n if not self.query or not self.query.strip():\n msg = \"JSON Query is required and cannot be blank.\"\n raise ValueError(msg)\n raw_data = self.get_data_dict()\n try:\n input_str = json.dumps(raw_data)\n repaired = repair_json(input_str)\n data_json = json.loads(repaired)\n jq_input = data_json[\"data\"] if isinstance(data_json, dict) and \"data\" in data_json else data_json\n results = jq.compile(self.query).input(jq_input).all()\n if not results:\n msg = \"No result from JSON query.\"\n raise ValueError(msg)\n result = results[0] if len(results) == 1 else results\n if result is None or result == \"None\":\n msg = \"JSON query returned null/None. Check if the path exists in your data.\"\n raise ValueError(msg)\n if isinstance(result, dict):\n return Data(data=result)\n return Data(data={\"result\": result})\n except (ValueError, TypeError, KeyError, json.JSONDecodeError) as e:\n logger.error(f\"JSON Query failed: {e}\")\n msg = f\"JSON Query error: {e}\"\n raise ValueError(msg) from e\n\n def get_normalized_data(self) -> dict:\n \"\"\"Get normalized data dictionary, handling the 'data' key if present.\"\"\"\n data_dict = self.get_data_dict()\n return data_dict.get(\"data\", data_dict)\n\n def data_is_list(self) -> bool:\n \"\"\"Check if data contains multiple items.\"\"\"\n return isinstance(self.data, list) and len(self.data) > 1\n\n def validate_single_data(self, operation: str) -> None:\n \"\"\"Validate that the operation is being performed on a single data object.\"\"\"\n if self.data_is_list():\n msg = f\"{operation} operation is not supported for multiple data objects.\"\n raise ValueError(msg)\n\n def operation_exception(self, operations: list[str]) -> None:\n \"\"\"Raise exception for incompatible operations.\"\"\"\n msg = f\"{operations} operations are not supported in combination with each other.\"\n raise ValueError(msg)\n\n # Data transformation operations\n def select_keys(self, *, evaluate: bool | None = None) -> Data:\n \"\"\"Select specific keys from the data dictionary.\"\"\"\n self.validate_single_data(\"Select Keys\")\n data_dict = self.get_normalized_data()\n filter_criteria: list[str] = self.select_keys_input\n\n # Filter the data\n if len(filter_criteria) == 1 and filter_criteria[0] == \"data\":\n filtered = data_dict[\"data\"]\n else:\n if not all(key in data_dict for key in filter_criteria):\n msg = f\"Select key not found in data. Available keys: {list(data_dict.keys())}\"\n raise ValueError(msg)\n filtered = {key: value for key, value in data_dict.items() if key in filter_criteria}\n\n # Create a new Data object with the filtered data\n if evaluate:\n filtered = self.recursive_eval(filtered)\n\n # Return a new Data object with the filtered data directly in the data attribute\n return Data(data=filtered)\n\n def remove_keys(self) -> Data:\n \"\"\"Remove specified keys from the data dictionary, recursively.\"\"\"\n self.validate_single_data(\"Remove Keys\")\n data_dict = self.get_normalized_data()\n remove_keys_input: list[str] = self.remove_keys_input\n\n filtered = DataOperationsComponent.remove_keys_recursive(data_dict, set(remove_keys_input))\n return Data(data=filtered)\n\n def rename_keys(self) -> Data:\n \"\"\"Rename keys in the data dictionary, recursively.\"\"\"\n self.validate_single_data(\"Rename Keys\")\n data_dict = self.get_normalized_data()\n rename_keys_input: dict[str, str] = self.rename_keys_input\n\n renamed = DataOperationsComponent.rename_keys_recursive(data_dict, rename_keys_input)\n return Data(data=renamed)\n\n def recursive_eval(self, data: Any) -> Any:\n \"\"\"Recursively evaluate string values in a dictionary or list.\n\n If the value is a string that can be evaluated, it will be evaluated.\n Otherwise, the original value is returned.\n \"\"\"\n if isinstance(data, dict):\n return {k: self.recursive_eval(v) for k, v in data.items()}\n if isinstance(data, list):\n return [self.recursive_eval(item) for item in data]\n if isinstance(data, str):\n try:\n # Only attempt to evaluate strings that look like Python literals\n if (\n data.strip().startswith((\"{\", \"[\", \"(\", \"'\", '\"'))\n or data.strip().lower() in (\"true\", \"false\", \"none\")\n or data.strip().replace(\".\", \"\").isdigit()\n ):\n return ast.literal_eval(data)\n # return data\n except (ValueError, SyntaxError, TypeError, MemoryError):\n # If evaluation fails for any reason, return the original string\n return data\n else:\n return data\n return data\n\n def evaluate_data(self) -> Data:\n \"\"\"Evaluate string values in the data dictionary.\"\"\"\n self.validate_single_data(\"Literal Eval\")\n logger.info(\"evaluating data\")\n return Data(**self.recursive_eval(self.get_data_dict()))\n\n def combine_data(self, *, evaluate: bool | None = None) -> Data:\n \"\"\"Combine multiple data objects into one.\"\"\"\n logger.info(\"combining data\")\n if not self.data_is_list():\n return self.data[0] if self.data else Data(data={})\n\n if len(self.data) == 1:\n msg = \"Combine operation requires multiple data inputs.\"\n raise ValueError(msg)\n\n data_dicts = [data.model_dump().get(\"data\", data.model_dump()) for data in self.data]\n combined_data = {}\n\n for data_dict in data_dicts:\n for key, value in data_dict.items():\n if key not in combined_data:\n combined_data[key] = value\n elif isinstance(combined_data[key], list):\n if isinstance(value, list):\n combined_data[key].extend(value)\n else:\n combined_data[key].append(value)\n else:\n # If current value is not a list, convert it to list and add new value\n combined_data[key] = (\n [combined_data[key], value] if not isinstance(value, list) else [combined_data[key], *value]\n )\n\n if evaluate:\n combined_data = self.recursive_eval(combined_data)\n\n return Data(**combined_data)\n\n def filter_data(self, input_data: list[dict[str, Any]], filter_key: str, filter_value: str, operator: str) -> list:\n \"\"\"Filter list data based on key, value, and operator.\"\"\"\n # Validate inputs\n if not input_data:\n self.status = \"Input data is empty.\"\n return []\n\n if not filter_key or not filter_value:\n self.status = \"Filter key or value is missing.\"\n return input_data\n\n # Filter the data\n filtered_data = []\n for item in input_data:\n if isinstance(item, dict) and filter_key in item:\n if self.compare_values(item[filter_key], filter_value, operator):\n filtered_data.append(item)\n else:\n self.status = f\"Warning: Some items don't have the key '{filter_key}' or are not dictionaries.\"\n\n return filtered_data\n\n def compare_values(self, item_value: Any, filter_value: str, operator: str) -> bool:\n comparison_func = OPERATORS.get(operator)\n if comparison_func:\n return comparison_func(item_value, filter_value)\n return False\n\n def multi_filter_data(self) -> Data:\n \"\"\"Apply multiple filters to the data.\"\"\"\n self.validate_single_data(\"Filter Values\")\n data_filtered = self.get_normalized_data()\n\n for filter_key in self.filter_key:\n if filter_key not in data_filtered:\n msg = f\"Filter key '{filter_key}' not found in data. Available keys: {list(data_filtered.keys())}\"\n raise ValueError(msg)\n\n if isinstance(data_filtered[filter_key], list):\n for filter_data in self.filter_values:\n filter_value = self.filter_values.get(filter_data)\n if filter_value is not None:\n data_filtered[filter_key] = self.filter_data(\n input_data=data_filtered[filter_key],\n filter_key=filter_data,\n filter_value=filter_value,\n operator=self.operator,\n )\n else:\n msg = f\"Filter key '{filter_key}' is not a list.\"\n raise TypeError(msg)\n\n return Data(**data_filtered)\n\n def append_update(self) -> Data:\n \"\"\"Append or Update with new key-value pairs.\"\"\"\n self.validate_single_data(\"Append or Update\")\n data_filtered = self.get_normalized_data()\n\n for key, value in self.append_update_data.items():\n data_filtered[key] = value\n\n return Data(**data_filtered)\n\n # Configuration and execution methods\n def update_build_config(self, build_config: dotdict, field_value: Any, field_name: str | None = None) -> dotdict:\n if field_name == \"operations\":\n build_config[\"operations\"][\"value\"] = field_value\n selected_actions = [action[\"name\"] for action in field_value]\n if len(selected_actions) == 1 and selected_actions[0] in ACTION_CONFIG:\n action = selected_actions[0]\n config = ACTION_CONFIG[action]\n build_config[\"data\"][\"is_list\"] = config[\"is_list\"]\n logger.info(config[\"log_msg\"])\n return set_current_fields(\n build_config=build_config,\n action_fields=self.actions_data,\n selected_action=action,\n default_fields=[\"operations\", \"data\"],\n func=set_field_display,\n )\n\n if field_name == \"mapped_json_display\":\n try:\n parsed_json = json.loads(field_value)\n keys = DataOperationsComponent.extract_all_paths(parsed_json)\n build_config[\"selected_key\"][\"options\"] = keys\n build_config[\"selected_key\"][\"show\"] = True\n except (json.JSONDecodeError, TypeError, ValueError) as e:\n logger.error(f\"Error parsing mapped JSON: {e}\")\n build_config[\"selected_key\"][\"show\"] = False\n\n return build_config\n\n def json_path(self) -> Data:\n try:\n if not self.data or not self.selected_key:\n msg = \"Missing input data or selected key.\"\n raise ValueError(msg)\n input_payload = self.data[0].data if isinstance(self.data, list) else self.data.data\n compiled = jq.compile(self.selected_key)\n result = compiled.input(input_payload).first()\n if isinstance(result, dict):\n return Data(data=result)\n return Data(data={\"result\": result})\n except (ValueError, TypeError, KeyError) as e:\n self.status = f\"Error: {e!s}\"\n self.log(self.status)\n return Data(data={\"error\": str(e)})\n\n def as_data(self) -> Data:\n if not hasattr(self, \"operations\") or not self.operations:\n return Data(data={})\n\n selected_actions = [action[\"name\"] for action in self.operations]\n logger.info(f\"selected_actions: {selected_actions}\")\n if len(selected_actions) != 1:\n return Data(data={})\n\n action_map: dict[str, Callable[[], Data]] = {\n \"Select Keys\": self.select_keys,\n \"Literal Eval\": self.evaluate_data,\n \"Combine\": self.combine_data,\n \"Filter Values\": self.multi_filter_data,\n \"Append or Update\": self.append_update,\n \"Remove Keys\": self.remove_keys,\n \"Rename Keys\": self.rename_keys,\n \"Path Selection\": self.json_path,\n \"JQ Expression\": self.json_query,\n }\n handler: Callable[[], Data] | None = action_map.get(selected_actions[0])\n if handler:\n try:\n return handler()\n except Exception as e:\n logger.error(f\"Error executing {selected_actions[0]}: {e!s}\")\n raise\n return Data(data={})\n" | ||
| "value": "import ast\nimport json\nfrom typing import TYPE_CHECKING, Any\n\nimport jq\nfrom json_repair import repair_json\n\nfrom lfx.custom import Component\nfrom lfx.inputs import DictInput, DropdownInput, MessageTextInput, SortableListInput\nfrom lfx.io import DataInput, MultilineInput, Output\nfrom lfx.log.logger import logger\nfrom lfx.schema import Data\nfrom lfx.schema.dotdict import dotdict\nfrom lfx.utils.component_utils import set_current_fields, set_field_display\n\nif TYPE_CHECKING:\n from collections.abc import Callable\n\nACTION_CONFIG = {\n \"Select Keys\": {\"is_list\": False, \"log_msg\": \"setting filter fields\"},\n \"Literal Eval\": {\"is_list\": False, \"log_msg\": \"setting evaluate fields\"},\n \"Combine\": {\"is_list\": True, \"log_msg\": \"setting combine fields\"},\n \"Filter Values\": {\"is_list\": False, \"log_msg\": \"setting filter values fields\"},\n \"Append or Update\": {\"is_list\": False, \"log_msg\": \"setting Append or Update fields\"},\n \"Remove Keys\": {\"is_list\": False, \"log_msg\": \"setting remove keys fields\"},\n \"Rename Keys\": {\"is_list\": False, \"log_msg\": \"setting rename keys fields\"},\n \"Path Selection\": {\"is_list\": False, \"log_msg\": \"setting mapped key extractor fields\"},\n \"JQ Expression\": {\"is_list\": False, \"log_msg\": \"setting parse json fields\"},\n}\nOPERATORS = {\n \"equals\": lambda a, b: str(a) == str(b),\n \"not equals\": lambda a, b: str(a) != str(b),\n \"contains\": lambda a, b: str(b) in str(a),\n \"starts with\": lambda a, b: str(a).startswith(str(b)),\n \"ends with\": lambda a, b: str(a).endswith(str(b)),\n}\n\n\nclass DataOperationsComponent(Component):\n display_name = \"Data Operations\"\n description = \"Perform various operations on a Data object.\"\n icon = \"file-json\"\n name = \"DataOperations\"\n default_keys = [\"operations\", \"data\"]\n metadata = {\n \"keywords\": [\n \"data\",\n \"operations\",\n \"filter values\",\n \"Append or Update\",\n \"remove keys\",\n \"rename keys\",\n \"select keys\",\n \"literal eval\",\n \"combine\",\n \"filter\",\n \"append\",\n \"update\",\n \"remove\",\n \"rename\",\n \"data operations\",\n \"data manipulation\",\n \"data transformation\",\n \"data filtering\",\n \"data selection\",\n \"data combination\",\n \"Parse JSON\",\n \"JSON Query\",\n \"JQ Query\",\n ],\n }\n actions_data = {\n \"Select Keys\": [\"select_keys_input\", \"operations\"],\n \"Literal Eval\": [],\n \"Combine\": [],\n \"Filter Values\": [\"filter_values\", \"operations\", \"operator\", \"filter_key\"],\n \"Append or Update\": [\"append_update_data\", \"operations\"],\n \"Remove Keys\": [\"remove_keys_input\", \"operations\"],\n \"Rename Keys\": [\"rename_keys_input\", \"operations\"],\n \"Path Selection\": [\"mapped_json_display\", \"selected_key\", \"operations\"],\n \"JQ Expression\": [\"query\", \"operations\"],\n }\n\n # All operation-specific input fields (used to hide and reset when no operation selected).\n ALL_OPERATION_FIELDS = [\n \"select_keys_input\",\n \"filter_key\",\n \"operator\",\n \"filter_values\",\n \"append_update_data\",\n \"remove_keys_input\",\n \"rename_keys_input\",\n \"mapped_json_display\",\n \"selected_key\",\n \"query\",\n ]\n\n @staticmethod\n def extract_all_paths(obj, path=\"\"):\n paths = []\n if isinstance(obj, dict):\n for k, v in obj.items():\n new_path = f\"{path}.{k}\" if path else f\".{k}\"\n paths.append(new_path)\n paths.extend(DataOperationsComponent.extract_all_paths(v, new_path))\n elif isinstance(obj, list) and obj:\n new_path = f\"{path}[0]\"\n paths.append(new_path)\n paths.extend(DataOperationsComponent.extract_all_paths(obj[0], new_path))\n return paths\n\n @staticmethod\n def remove_keys_recursive(obj, keys_to_remove):\n if isinstance(obj, dict):\n return {\n k: DataOperationsComponent.remove_keys_recursive(v, keys_to_remove)\n for k, v in obj.items()\n if k not in keys_to_remove\n }\n if isinstance(obj, list):\n return [DataOperationsComponent.remove_keys_recursive(item, keys_to_remove) for item in obj]\n return obj\n\n @staticmethod\n def rename_keys_recursive(obj, rename_map):\n if isinstance(obj, dict):\n return {\n rename_map.get(k, k): DataOperationsComponent.rename_keys_recursive(v, rename_map)\n for k, v in obj.items()\n }\n if isinstance(obj, list):\n return [DataOperationsComponent.rename_keys_recursive(item, rename_map) for item in obj]\n return obj\n\n inputs = [\n DataInput(name=\"data\", display_name=\"Data\", info=\"Data object to filter.\", required=True, is_list=True),\n SortableListInput(\n name=\"operations\",\n display_name=\"Operations\",\n placeholder=\"Select Operation\",\n info=\"List of operations to perform on the data.\",\n options=[\n {\"name\": \"Select Keys\", \"icon\": \"lasso-select\"},\n {\"name\": \"Literal Eval\", \"icon\": \"braces\"},\n {\"name\": \"Combine\", \"icon\": \"merge\"},\n {\"name\": \"Filter Values\", \"icon\": \"filter\"},\n {\"name\": \"Append or Update\", \"icon\": \"circle-plus\"},\n {\"name\": \"Remove Keys\", \"icon\": \"eraser\"},\n {\"name\": \"Rename Keys\", \"icon\": \"pencil-line\"},\n {\"name\": \"Path Selection\", \"icon\": \"mouse-pointer\"},\n {\"name\": \"JQ Expression\", \"icon\": \"terminal\"},\n ],\n real_time_refresh=True,\n limit=1,\n ),\n # select keys inputs\n MessageTextInput(\n name=\"select_keys_input\",\n display_name=\"Select Keys\",\n info=\"List of keys to select from the data. Only top-level keys can be selected.\",\n show=False,\n is_list=True,\n value=[],\n ),\n # filter values inputs\n MessageTextInput(\n name=\"filter_key\",\n display_name=\"Filter Key\",\n info=(\n \"Name of the key containing the list to filter. \"\n \"It must be a top-level key in the JSON and its value must be a list.\"\n ),\n is_list=True,\n show=False,\n value=[],\n ),\n DropdownInput(\n name=\"operator\",\n display_name=\"Comparison Operator\",\n options=[\"equals\", \"not equals\", \"contains\", \"starts with\", \"ends with\"],\n info=\"The operator to apply for comparing the values.\",\n value=\"equals\",\n advanced=False,\n show=False,\n ),\n DictInput(\n name=\"filter_values\",\n display_name=\"Filter Values\",\n info=\"List of values to filter by.\",\n show=False,\n is_list=True,\n value={},\n ),\n # update/ Append data inputs\n DictInput(\n name=\"append_update_data\",\n display_name=\"Append or Update\",\n info=\"Data to append or update the existing data with. Only top-level keys are checked.\",\n show=False,\n value={\"key\": \"value\"},\n is_list=True,\n ),\n # remove keys inputs\n MessageTextInput(\n name=\"remove_keys_input\",\n display_name=\"Remove Keys\",\n info=\"List of keys to remove from the data.\",\n show=False,\n is_list=True,\n value=[],\n ),\n # rename keys inputs\n DictInput(\n name=\"rename_keys_input\",\n display_name=\"Rename Keys\",\n info=\"List of keys to rename in the data.\",\n show=False,\n is_list=True,\n value={\"old_key\": \"new_key\"},\n ),\n MultilineInput(\n name=\"mapped_json_display\",\n display_name=\"JSON to Map\",\n info=\"Paste or preview your JSON here to explore its structure and select a path for extraction.\",\n required=False,\n refresh_button=True,\n real_time_refresh=True,\n placeholder=\"Add a JSON example.\",\n show=False,\n ),\n DropdownInput(\n name=\"selected_key\",\n display_name=\"Select Path\",\n options=[],\n required=False,\n dynamic=True,\n show=False,\n value=None,\n ),\n MessageTextInput(\n name=\"query\",\n display_name=\"JQ Expression\",\n info=\"JSON Query to filter the data. Used by Parse JSON operation.\",\n placeholder=\"e.g., .properties.id\",\n show=False,\n ),\n ]\n\n # Default values for operation fields when clearing (match input definitions)\n OPERATION_FIELD_DEFAULTS: dict[str, Any] = {\n \"select_keys_input\": [],\n \"filter_key\": [],\n \"operator\": \"equals\",\n \"filter_values\": {},\n \"append_update_data\": {\"key\": \"value\"},\n \"remove_keys_input\": [],\n \"rename_keys_input\": {\"old_key\": \"new_key\"},\n \"mapped_json_display\": \"\",\n \"selected_key\": None,\n \"query\": \"\",\n }\n\n outputs = [\n Output(display_name=\"Data\", name=\"data_output\", method=\"as_data\"),\n ]\n\n # Helper methods for data operations\n def get_data_dict(self) -> dict:\n \"\"\"Extract data dictionary from Data object.\"\"\"\n data = self.data[0] if isinstance(self.data, list) and len(self.data) == 1 else self.data\n return data.model_dump()\n\n def json_query(self) -> Data:\n import json\n\n import jq\n\n if not self.query or not self.query.strip():\n msg = \"JSON Query is required and cannot be blank.\"\n raise ValueError(msg)\n raw_data = self.get_data_dict()\n try:\n input_str = json.dumps(raw_data)\n repaired = repair_json(input_str)\n data_json = json.loads(repaired)\n jq_input = data_json[\"data\"] if isinstance(data_json, dict) and \"data\" in data_json else data_json\n results = jq.compile(self.query).input(jq_input).all()\n if not results:\n msg = \"No result from JSON query.\"\n raise ValueError(msg)\n result = results[0] if len(results) == 1 else results\n if result is None or result == \"None\":\n msg = \"JSON query returned null/None. Check if the path exists in your data.\"\n raise ValueError(msg)\n if isinstance(result, dict):\n return Data(data=result)\n return Data(data={\"result\": result})\n except (ValueError, TypeError, KeyError, json.JSONDecodeError) as e:\n logger.error(f\"JSON Query failed: {e}\")\n msg = f\"JSON Query error: {e}\"\n raise ValueError(msg) from e\n\n def get_normalized_data(self) -> dict:\n \"\"\"Get normalized data dictionary, handling the 'data' key if present.\"\"\"\n data_dict = self.get_data_dict()\n return data_dict.get(\"data\", data_dict)\n\n def data_is_list(self) -> bool:\n \"\"\"Check if data contains multiple items.\"\"\"\n return isinstance(self.data, list) and len(self.data) > 1\n\n def validate_single_data(self, operation: str) -> None:\n \"\"\"Validate that the operation is being performed on a single data object.\"\"\"\n if self.data_is_list():\n msg = f\"{operation} operation is not supported for multiple data objects.\"\n raise ValueError(msg)\n\n def operation_exception(self, operations: list[str]) -> None:\n \"\"\"Raise exception for incompatible operations.\"\"\"\n msg = f\"{operations} operations are not supported in combination with each other.\"\n raise ValueError(msg)\n\n # Data transformation operations\n def select_keys(self, *, evaluate: bool | None = None) -> Data:\n \"\"\"Select specific keys from the data dictionary.\"\"\"\n self.validate_single_data(\"Select Keys\")\n data_dict = self.get_normalized_data()\n filter_criteria: list[str] = self.select_keys_input\n\n # Filter the data\n if len(filter_criteria) == 1 and filter_criteria[0] == \"data\":\n filtered = data_dict[\"data\"]\n else:\n if not all(key in data_dict for key in filter_criteria):\n msg = f\"Select key not found in data. Available keys: {list(data_dict.keys())}\"\n raise ValueError(msg)\n filtered = {key: value for key, value in data_dict.items() if key in filter_criteria}\n\n # Create a new Data object with the filtered data\n if evaluate:\n filtered = self.recursive_eval(filtered)\n\n # Return a new Data object with the filtered data directly in the data attribute\n return Data(data=filtered)\n\n def remove_keys(self) -> Data:\n \"\"\"Remove specified keys from the data dictionary, recursively.\"\"\"\n self.validate_single_data(\"Remove Keys\")\n data_dict = self.get_normalized_data()\n remove_keys_input: list[str] = self.remove_keys_input\n\n filtered = DataOperationsComponent.remove_keys_recursive(data_dict, set(remove_keys_input))\n return Data(data=filtered)\n\n def rename_keys(self) -> Data:\n \"\"\"Rename keys in the data dictionary, recursively.\"\"\"\n self.validate_single_data(\"Rename Keys\")\n data_dict = self.get_normalized_data()\n rename_keys_input: dict[str, str] = self.rename_keys_input\n\n renamed = DataOperationsComponent.rename_keys_recursive(data_dict, rename_keys_input)\n return Data(data=renamed)\n\n def recursive_eval(self, data: Any) -> Any:\n \"\"\"Recursively evaluate string values in a dictionary or list.\n\n If the value is a string that can be evaluated, it will be evaluated.\n Otherwise, the original value is returned.\n \"\"\"\n if isinstance(data, dict):\n return {k: self.recursive_eval(v) for k, v in data.items()}\n if isinstance(data, list):\n return [self.recursive_eval(item) for item in data]\n if isinstance(data, str):\n try:\n # Only attempt to evaluate strings that look like Python literals\n if (\n data.strip().startswith((\"{\", \"[\", \"(\", \"'\", '\"'))\n or data.strip().lower() in (\"true\", \"false\", \"none\")\n or data.strip().replace(\".\", \"\").isdigit()\n ):\n return ast.literal_eval(data)\n # return data\n except (ValueError, SyntaxError, TypeError, MemoryError):\n # If evaluation fails for any reason, return the original string\n return data\n else:\n return data\n return data\n\n def evaluate_data(self) -> Data:\n \"\"\"Evaluate string values in the data dictionary.\"\"\"\n self.validate_single_data(\"Literal Eval\")\n logger.info(\"evaluating data\")\n return Data(**self.recursive_eval(self.get_data_dict()))\n\n def combine_data(self, *, evaluate: bool | None = None) -> Data:\n \"\"\"Combine multiple data objects into one.\"\"\"\n logger.info(\"combining data\")\n if not self.data_is_list():\n return self.data[0] if self.data else Data(data={})\n\n if len(self.data) == 1:\n msg = \"Combine operation requires multiple data inputs.\"\n raise ValueError(msg)\n\n data_dicts = [data.model_dump().get(\"data\", data.model_dump()) for data in self.data]\n combined_data = {}\n\n for data_dict in data_dicts:\n for key, value in data_dict.items():\n if key not in combined_data:\n combined_data[key] = value\n elif isinstance(combined_data[key], list):\n if isinstance(value, list):\n combined_data[key].extend(value)\n else:\n combined_data[key].append(value)\n else:\n # If current value is not a list, convert it to list and add new value\n combined_data[key] = (\n [combined_data[key], value] if not isinstance(value, list) else [combined_data[key], *value]\n )\n\n if evaluate:\n combined_data = self.recursive_eval(combined_data)\n\n return Data(**combined_data)\n\n def filter_data(self, input_data: list[dict[str, Any]], filter_key: str, filter_value: str, operator: str) -> list:\n \"\"\"Filter list data based on key, value, and operator.\"\"\"\n # Validate inputs\n if not input_data:\n self.status = \"Input data is empty.\"\n return []\n\n if not filter_key or not filter_value:\n self.status = \"Filter key or value is missing.\"\n return input_data\n\n # Filter the data\n filtered_data = []\n for item in input_data:\n if isinstance(item, dict) and filter_key in item:\n if self.compare_values(item[filter_key], filter_value, operator):\n filtered_data.append(item)\n else:\n self.status = f\"Warning: Some items don't have the key '{filter_key}' or are not dictionaries.\"\n\n return filtered_data\n\n def compare_values(self, item_value: Any, filter_value: str, operator: str) -> bool:\n comparison_func = OPERATORS.get(operator)\n if comparison_func:\n return comparison_func(item_value, filter_value)\n return False\n\n def multi_filter_data(self) -> Data:\n \"\"\"Apply multiple filters to the data.\"\"\"\n self.validate_single_data(\"Filter Values\")\n data_filtered = self.get_normalized_data()\n\n for filter_key in self.filter_key:\n if filter_key not in data_filtered:\n msg = f\"Filter key '{filter_key}' not found in data. Available keys: {list(data_filtered.keys())}\"\n raise ValueError(msg)\n\n if isinstance(data_filtered[filter_key], list):\n for filter_data in self.filter_values:\n filter_value = self.filter_values.get(filter_data)\n if filter_value is not None:\n data_filtered[filter_key] = self.filter_data(\n input_data=data_filtered[filter_key],\n filter_key=filter_data,\n filter_value=filter_value,\n operator=self.operator,\n )\n else:\n msg = f\"Filter key '{filter_key}' is not a list.\"\n raise TypeError(msg)\n\n return Data(**data_filtered)\n\n def append_update(self) -> Data:\n \"\"\"Append or Update with new key-value pairs.\"\"\"\n self.validate_single_data(\"Append or Update\")\n data_filtered = self.get_normalized_data()\n\n for key, value in self.append_update_data.items():\n data_filtered[key] = value\n\n return Data(**data_filtered)\n\n # Configuration and execution methods\n def update_build_config(self, build_config: dotdict, field_value: Any, field_name: str | None = None) -> dotdict:\n if field_name == \"operations\":\n build_config[\"operations\"][\"value\"] = field_value\n # Mirror Text Operations: first hide all operation-specific fields and clear their values\n for field in self.ALL_OPERATION_FIELDS:\n if field in build_config:\n build_config[field][\"show\"] = False\n if field in self.OPERATION_FIELD_DEFAULTS:\n build_config[field][\"value\"] = self.OPERATION_FIELD_DEFAULTS[field]\n\n selected_actions = [\n action[\"name\"] for action in (field_value or []) if isinstance(action, dict) and \"name\" in action\n ]\n if len(selected_actions) == 1 and selected_actions[0] in ACTION_CONFIG:\n action = selected_actions[0]\n config = ACTION_CONFIG[action]\n build_config[\"data\"][\"is_list\"] = config[\"is_list\"]\n logger.info(config[\"log_msg\"])\n return set_current_fields(\n build_config=build_config,\n action_fields=self.actions_data,\n selected_action=action,\n default_fields=[\"operations\", \"data\"],\n func=set_field_display,\n )\n return build_config\n\n if field_name == \"mapped_json_display\":\n try:\n parsed_json = json.loads(field_value)\n keys = DataOperationsComponent.extract_all_paths(parsed_json)\n build_config[\"selected_key\"][\"options\"] = keys\n build_config[\"selected_key\"][\"show\"] = True\n except (json.JSONDecodeError, TypeError, ValueError) as e:\n logger.error(f\"Error parsing mapped JSON: {e}\")\n build_config[\"selected_key\"][\"show\"] = False\n\n return build_config\n\n def json_path(self) -> Data:\n try:\n if not self.data or not self.selected_key:\n msg = \"Missing input data or selected key.\"\n raise ValueError(msg)\n input_payload = self.data[0].data if isinstance(self.data, list) else self.data.data\n compiled = jq.compile(self.selected_key)\n result = compiled.input(input_payload).first()\n if isinstance(result, dict):\n return Data(data=result)\n return Data(data={\"result\": result})\n except (ValueError, TypeError, KeyError) as e:\n self.status = f\"Error: {e!s}\"\n self.log(self.status)\n return Data(data={\"error\": str(e)})\n\n def as_data(self) -> Data:\n if not hasattr(self, \"operations\") or not self.operations:\n return Data(data={})\n\n selected_actions = [action[\"name\"] for action in self.operations]\n logger.info(f\"selected_actions: {selected_actions}\")\n if len(selected_actions) != 1:\n return Data(data={})\n\n action_map: dict[str, Callable[[], Data]] = {\n \"Select Keys\": self.select_keys,\n \"Literal Eval\": self.evaluate_data,\n \"Combine\": self.combine_data,\n \"Filter Values\": self.multi_filter_data,\n \"Append or Update\": self.append_update,\n \"Remove Keys\": self.remove_keys,\n \"Rename Keys\": self.rename_keys,\n \"Path Selection\": self.json_path,\n \"JQ Expression\": self.json_query,\n }\n handler: Callable[[], Data] | None = action_map.get(selected_actions[0])\n if handler:\n try:\n return handler()\n except Exception as e:\n logger.error(f\"Error executing {selected_actions[0]}: {e!s}\")\n raise\n return Data(data={})\n" |
There was a problem hiding this comment.
Inconsistent defensiveness when extracting selected_actions in as_data vs update_build_config.
In update_build_config, the extraction of action names is defensive:
selected_actions = [
action["name"] for action in (field_value or []) if isinstance(action, dict) and "name" in action
]But in as_data, the extraction assumes every item in self.operations is a dict with a "name" key:
selected_actions = [action["name"] for action in self.operations]If self.operations ever contains a malformed entry (e.g., a string or a dict without "name"), as_data will raise a KeyError/TypeError at runtime. Consider using the same guard pattern.
Proposed fix
In the embedded as_data method (within the JSON "value" string and the corresponding .py source):
- selected_actions = [action["name"] for action in self.operations]
+ selected_actions = [
+ action["name"] for action in self.operations if isinstance(action, dict) and "name" in action
+ ]🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/lfx/src/lfx/_assets/component_index.json` at line 95583, The as_data
method assumes every item in self.operations is a dict with a "name" key which
can raise KeyError/TypeError; change the selected_actions extraction in as_data
to the same defensive pattern used in update_build_config (filtering with
isinstance(action, dict) and "name" in action) when building selected_actions
from self.operations, ensuring you handle empty/malformed entries gracefully
before looking up action_map and invoking handlers.
|
|
||
| # Default values for operation fields when clearing (match input definitions) | ||
| OPERATION_FIELD_DEFAULTS: dict[str, Any] = { | ||
| "select_keys_input": [], | ||
| "filter_key": [], | ||
| "operator": "equals", | ||
| "filter_values": {}, | ||
| "append_update_data": {"key": "value"}, | ||
| "remove_keys_input": [], | ||
| "rename_keys_input": {"old_key": "new_key"}, | ||
| "mapped_json_display": "", | ||
| "selected_key": None, | ||
| "query": "", | ||
| } |
There was a problem hiding this comment.
Mutable default values shared by reference — will corrupt future resets.
OPERATION_FIELD_DEFAULTS contains mutable objects ([], {}, {"key": "value"}). The assignment on line 503:
build_config[field]["value"] = self.OPERATION_FIELD_DEFAULTS[field]binds the same list/dict instance to build_config. If any downstream code mutates that value in-place (e.g., .append(), [key] = ...), it permanently corrupts the class-level defaults for all subsequent calls.
Fix: deep-copy defaults on assignment
+from copy import deepcopy
...
for field in self.ALL_OPERATION_FIELDS:
if field in build_config:
build_config[field]["show"] = False
if field in self.OPERATION_FIELD_DEFAULTS:
- build_config[field]["value"] = self.OPERATION_FIELD_DEFAULTS[field]
+ build_config[field]["value"] = deepcopy(self.OPERATION_FIELD_DEFAULTS[field])Alternatively, make OPERATION_FIELD_DEFAULTS a method that returns fresh copies each time.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/lfx/src/lfx/components/processing/data_operations.py` around lines 248 -
261, OPERATION_FIELD_DEFAULTS currently contains mutable shared instances and
the assignment build_config[field]["value"] =
self.OPERATION_FIELD_DEFAULTS[field] binds those shared objects; change the
assignment to provide a fresh copy (e.g., use
copy.deepcopy(self.OPERATION_FIELD_DEFAULTS[field]) or replace
OPERATION_FIELD_DEFAULTS with a function like get_operation_field_defaults()
that returns new dict/list instances) so each build_config gets its own
independent mutable objects and class-level defaults are not mutated by
consumers.
The Vite frontend build was configured with --max-old-space-size=12288 (12GB), which exceeds available RAM on ARM64 CI runners, causing the build process to be OOM-killed during the transform phase. Reduced to 4GB (4096MB) which is sufficient for the Vite build and prevents OOM kills in memory-constrained Docker BuildKit environments.
The recursive chown -R on /app was re-owning the entire .venv (~2.6GB, 40k+ files) which was already correctly owned via COPY --chown=1000:0. This was causing the build to be killed on ARM64 runners. Changed to non-recursive chown on /app since only the directory itself needs ownership set. /app/data still gets recursive chown (it's empty).
The 40GB ARM64 runner runs out of disk when building 3 Docker images sequentially. Each image (main ~8GB layers, backend ~5GB, frontend) accumulates build cache and layers that exhaust the disk. Added cleanup steps between builds that: - Remove the tested image (no longer needed) - Prune all unused Docker data and buildx cache - Log disk usage before/after for debugging
Description
The "Data Operations" component fails to reset its UI state when an operation is removed. When a user selects an operation (e.g., "Filter Values"), the component correctly displays the necessary input fields. However, clicking the "X" to remove the operation only removes the operation's name, leaving the associated input fields (Filter Key, Comparison Operator, etc.) orphaned on the component UI. In contrast, the "Text Operations" component (which shares a similar logic) behaves as expected by clearing all inputs upon removal.
Steps to reproduce
operations.mov
Summary by CodeRabbit
Release Notes
New Features
Tests