Goal

Enable Parameter Providers to automatically refresh parameter values from external sources at configurable intervals, allowing NiFi flows to automatically adapt to rotating secrets, updated configuration values, and other externally managed parameters without manual intervention.

Background and Current State

Parameter Providers Overview

Parameter Providers were introduced in Apache NiFi 1.18.0 as extension points that allow parameters to be fetched from external sources such as:

  • HashiCorp Vault secrets engines
  • AWS Secrets Manager
  • Google Cloud Secret Manager
  • File-based systems (e.g., Kubernetes secrets)
  • Environment variables

Parameter Providers enable automatic creation and population of Parameter Contexts from these external sources, providing powerful configurability and portability for NiFi flows.

Current Manual Operation

The current Parameter Provider workflow requires manual intervention:

  1. Initial Setup: Users configure a Parameter Provider and map parameter groups to Parameter Contexts
  2. Manual Refresh: When external parameter values change (e.g., secret rotation), users must:
    • Navigate to Controller Settings → Parameter Providers tab
    • Click "Fetch Parameters" button for the specific provider
    • Review the "Apply Fetched Parameters" dialog showing changed values
    • Manually apply changes, which triggers restart of affected components

Note that new values would also be retrieved in case of a NiFi restart. The parameter values coming through a parameter provider are NOT persisted by NiFi and are kept in memory. In case of restart, NiFi would fetch again all of the values for existing parameters that are defined through parameter providers.

Limitations of Manual Approach

The manual approach creates operational overhead and delays in several scenarios:

  • Secret Rotation: Security policies often require regular rotation of secrets (passwords, API keys, certificates)
  • Configuration Updates: Environment-specific configuration changes made in external systems
  • Compliance Requirements: Automated secret rotation mandated by security frameworks
  • High Availability: Manual processes introduce potential downtime windows and human error

Current Workaround: NiFi CLI Scripting

Users can partially automate parameter updates using the NiFi CLI fetch-params command:

bash

# Fetch and apply parameters for a specific provider
./nifi.sh fetch-params --parameterProviderId <provider-id> --apply

# Script example for periodic execution
#!/bin/bash
PROVIDER_ID="vault-secrets-provider"
./nifi.sh fetch-params --parameterProviderId $PROVIDER_ID --apply
if [ $? -eq 0 ]; then
    echo "Parameters updated successfully"
else
    echo "Parameter update failed"
    # Handle error (alerting, logging, etc.)
fi

Limitations of CLI Approach:

  • Requires external orchestration (cron jobs, task schedulers)
  • No integration with NiFi's scheduling and monitoring infrastructure
  • Limited error handling and retry capabilities
  • Potential for external script failures or missed executions
  • No built-in audit logging of automatic parameter changes

Implementation Approaches

Approach 1: Framework-Level Scheduling Integration

Overview

Extend the existing NiFi scheduling framework to support Parameter Providers as schedulable components, similar to how Reporting Tasks are currently handled.

Key Requirements

  1. Backward Compatibility: Existing Parameter Providers must continue working unchanged
  2. Opt-in Behavior: Automatic refresh must be explicitly enabled (disabled by default)
  3. Selective Updates: Only update existing parameters, ignore new parameters (new parameters may be sensitive or not, and it is better to leave this decision to a user)
  4. Audit Trail: Log all automatic parameter changes for security and compliance
  5. Error Handling: Graceful handling of external system failures
  6. Schedule Flexibility: Support both timer-driven and CRON-driven scheduling

High-Level Architecture Changes


1. Framework Scheduler Integration

  • Extend FlowController to manage Parameter Provider scheduling
  • Integrate with existing ProcessScheduler for unified scheduling management
  • Add Parameter Provider nodes to the scheduling engine alongside Processors and Reporting Tasks

2. Node-Level Scheduling (Following ReportingTaskNode Pattern)

java

public interface ParameterProviderNode extends ComponentNode {
    // Scheduling methods - identical to ReportingTaskNode
    void setSchedulingStrategy(SchedulingStrategy schedulingStrategy);
    SchedulingStrategy getSchedulingStrategy();
    void setSchedulingPeriod(String schedulingPeriod);
    String getSchedulingPeriod();
    
    // Parameter Provider specific settings
    void setAutoApplyChanges(boolean autoApply);
    boolean isAutoApplyChanges();
    
    // Standard node methods
    ParameterProvider getParameterProvider();
    void verifyCanSchedule();
    void verifyCanStop();
}

3. Framework Integration (Mirror ReportingTask Architecture)

The default value for scheduling period would be "0 sec". This specific default would be used as a way to disable the automatic refresh of the parameter values and would ensure backward compatibility.

java

public class ParameterProviderNode {
    private boolean automaticRefreshEnabled = false;  // Default: disabled
    private SchedulingStrategy schedulingStrategy = SchedulingStrategy.TIMER_DRIVEN;
    private String schedulingPeriod = "0 sec";        // Default: disabled (special value)
    private boolean autoApplyChanges = false;         // Default: manual approval required
}

4. UI Components

  • Add scheduling configuration section to Parameter Provider dialog (identical to ReportingTask dialog)
  • Include enable/disable through "0 sec" scheduling period convention
  • Provide scheduling strategy and period configuration (same UI components as ReportingTask)
  • Add auto-apply changes option with appropriate warnings

5. REST API Extensions

  • Extend Parameter Provider endpoints to include scheduling configuration
  • Add validation for scheduling parameters
  • Include scheduling state in Parameter Provider status responses

Required Code Changes

Core Framework:

  • nifi-framework-core: Scheduler integration identical to ReportingTask handling
  • nifi-framework-api: ParameterProviderNode interface (mirrors ReportingTaskNode)
  • nifi-web-api: REST endpoint extensions using same patterns as ReportingTask endpoints
  • nifi-web-ui: UI components reusing ReportingTask scheduling components

Parameter Provider Infrastructure:

  • StandardParameterProviderNode: Implementation mirroring StandardReportingTaskNode
  • Parameter Provider implementations: Use standard lifecycle annotations (@OnScheduled, @OnUnscheduled)
  • Configuration persistence: Database schema updates identical to ReportingTask scheduling properties

Pros

  • Architectural Consistency: Follows established patterns used by Reporting Tasks
  • Unified Scheduling: Leverages existing, proven scheduling infrastructure
  • Resource Management: Centralized thread and resource management
  • UI Consistency: Users familiar with Reporting Task scheduling will understand Parameter Provider scheduling
  • Framework Integration: Built-in error handling, monitoring, and lifecycle management

Cons

  • Implementation Complexity: Requires changes across multiple framework modules

Approach 2: Provider-Level Background Threading

Overview

Implement automatic refresh capability within the AbstractParameterProvider class using provider-managed background threads, without modifying the core scheduling framework.

High-Level Architecture Changes

1. Abstract Provider Enhancement

java

public abstract class AbstractParameterProvider implements ParameterProvider {
    private boolean autoRefreshEnabled = false;
    private String refreshInterval = "0 sec";
    private ScheduledExecutorService scheduledExecutor;
    private volatile boolean running = false;
    
    protected void startAutoRefresh() {
        if (autoRefreshEnabled && !"0 sec".equals(refreshInterval)) {
            // Start background thread for periodic refresh
        }
    }
    
    protected abstract void onAutoRefresh();
}

2. Provider-Specific Configuration

  • Add refresh-related properties to each Parameter Provider
  • Implement background thread management in AbstractParameterProvider
  • Handle thread lifecycle (start/stop/cleanup) during provider enable/disable

3. Thread Management

  • Each Parameter Provider manages its own ScheduledExecutorService
  • Implement proper thread cleanup during component shutdown
  • Handle thread interruption and graceful stopping

Required Code Changes

Parameter Provider Module:

  • AbstractParameterProvider: Add background threading and refresh logic
  • Provider implementations: Override refresh behavior if needed
  • Property descriptors: Add refresh interval and auto-apply configuration

Configuration Persistence:

  • Property-based configuration (no schema changes required)
  • Standard PropertyDescriptor validation for refresh intervals

Pros

  • Focused Scope: Changes contained within Parameter Provider module
  • Faster Implementation: Lower development effort
  • Provider Autonomy: Each provider can customize its refresh behavior
  • Minimal Framework Impact: No changes to core scheduling infrastructure

Cons

  • Architectural Inconsistency: Deviates from established NiFi patterns for scheduled components
  • Resource Management Complexity: Multiple providers create multiple thread pools
  • Threading Overhead: Potential resource waste with many providers running separate threads - very unlikely though
  • Lifecycle Management: More complex shutdown and cleanup procedures


  • No labels