C2 Protocol Introduction

Command and control, herein C2, consists of a C2 server and C2 agents. MiNIFi agents must adhere to the C2 protocols to have successful communications. C2 communications occur over a variety of protocols. Currently an HTTP/HTTPS RESTFul paradigm exists to support C2 capabilities to MiNiFi C2 agents. In the future additional protocols may become available for use. Note that when the phrasing "C2 designator" is used, this implies the C2 server, server, or agent that is designated as a responder to the C2 agent(s). All protocols must support the following operations:


Operation NameDescription

ACKNOWLEDGE

Operation used by MiNiFi C2 agents to acknowledge the receipt and execution of a C2 server requested operation

CLEAR

Clears flow connection queues or repositories on the C2 agent

DESCRIBE

Currently Unused

HEARTBEAT

Heartbeat provides status and operational capabilities to C2 server(s)
PAUSEPauses the execution of flows on the C2 agent

RESTART

Restarts C2 agents
RESUMEResumes the execution of flows on the C2 agent

START

Starts components within the C2 agents
STOP Stops components within the C2 agent
TRANSFER

Transfers an object between the C2 agent and C2 designator.

UPDATEUpdates components of the C2 agent or the flow configuration.

C2 Requirements

The requirements are an evolving list that have grown organically from an implementation. Any other portions of a heartbeat are considered optional.

C2 RequirementJustification and Purpose
C2 agents shall report C2 status at regular intervals through a heartbeat messageAgents must employ heartbeat messages that follow an interval that is favorable to the agent ( power )
C2 agents shall report the flow version within the heartbeat message at regular intervalsAgents must report flow version to the C2 server
C2 agents shall report queue status within the heartbeat message at regular intervalsAgents must report queue status to the C2 server at regular intervals
C2 agents shall execute acknowledge commands sent via a heartbeat responseAgents must execute and acknowledge commands from the C2 server
C2 agents shall apply requested changes and inform the C2 server of success or failureAgents must apply and acknowledge updates from the C2 server, responding with a success or failure
C2 agents shall implement clear, update, restart, start, stop, and transfer commandsAgents must implement the prescribed commands.

C2 Messages 


Heartbeats

   Primary communications are carried over a C2 heartbeat. The heartbeat contains operational information about the C2 agent and can occur at a configurable frequency. The heartbeat provides status information to the C2 server. The response from the heartbeat contains requested operations from the C2 server. These operations are then acknowledged if/when they are completed. This means that the heartbeat is the only operation initiated by the C2 agent and the C2 server responds directly to these heartbeats. Version four of the heartbeat will allow a subscription model to be used for heartbeats to avoid sending unnecessary information. Though Describe can provide parts of the same information the aggregate produced for a heartbeat allows the agent to flush messaging queues to ensure subscribed heartbeats have the most up to date information. Heartbeats are intended to be lightweight structures with minimal information; however, the subscription model supports differing wire protocols and deployment scenarios that support larger payloads.

Protocols

HTTP/S Protocol

  The HTTP/S protocol supports a url for heartbeating and acknowledging operations. These endpoints support the JSON structures defined below. C2 agents must send a heartbeat, defined above, to update the C2 server of its status and to receive operations. The frequency of these calls are up to the C2 agent to define. 

CoAP Protocol

  The CoAP protocol (https://coap.technology/) supports a constrained protocol for smaller devices. In the case of CoAP, the base requirements, as listed above, are fulfilled in each message. While the heartbeat structure, below, contains optional elements, the CoAP protocol implemented in Apache NiFi MiNiFi C++ contains minimal information per heartbeat.

Heartbeat structure

Heartbeats consist of a POST of the following Schema to the C2 heartbeat url. Metrics is a configurable list of metrics that can be returned, so the entirety of that object is optional. AvailableClasses in the structure below is optional.

Update: current work uses a flow identifier to help identify the currently running flow within DeviceInfo. 

Key Value
agentInfo

Key Value
agentClass <string>
agentManifest

Key Value
buildInfo

Key Value
compiler <string>
flags <string>
revision <string>
timestamp <int>
version <string>

bundles

artifact componentManifest version group
artifact name <string>

Key Value
controllerServices

supportsDynamicProperties typeDescription propertyDescriptors type supportsDynamicRelationships
<true/false> <string>

Key Value
Property1

Key Value
defaultValue <value>
description <string>
expressionLanguageScope <string>
name <string>
required <true/false>
validator string

ControllerService name <string> <true/false>

processors

supportedRelationships isSingleThreaded supportsDynamicRelationships supportsDynamicProperties typeDescription propertyDescriptors type inputRequirement

name description
success success
failure failure

<true/false> <true/false> <true/false> <string>

Key Value
Property1

Key Value
defaultValue <value>
description <string>
expressionLanguageScope <string>
name <string>
required <true/false>
validator string

Processor name <string> <INPUT_FORBIDDEN/INPUT_ALLOWED/INPUT_REQUIRED>

<string> <string>

agentManifestHash <string>
identifier agent id <string>
status

Key Value
components

Key Value
componentN

Key Value
running <true/false>
uuid <uuid string>

repositories

Key Value
ff

Key Value
full <true/false>
running <true/false>
size <int>

repo_name

Key Value
full <true/false>
running <true/false>
size <int>

resourceConsumption

Key Value
cpuUtilization agent cpu usage since last heartbeat <float>
memoryUsage agent memory usage in bytes <int>

uptime <int>

deviceInfo

Key Value
identifier device id <string>
networkInfo

Key Value
hostname <string>
ipAddress <string>

systemInfo

Key Value
cpuUtilization device cpu usage since last heartbeat <float>
machinearch machine arch <string>
memoryUsage device memory usage in bytes <int>
operatingSystem <string>
physicalMem device memory in bytes <int>
vCores device vCores <int>

metrics

Key Value
QueueMetrics

Key Value
Connection

Key Value
datasize <string>
datasizemax <string>
queued <string>
queuedmax <string>

RepositoryMetrics

Key Value
flowfile

Key Value
full <true/false>
running <true/false>
size <string>

provenance

Key Value
full <true/false>
running <true/false>
size <string>

operation heartbeat
Responses to the heartbeats have the following structure
Key Value
Operation heartbeat
Requested_operations

Name Operationid Operation Content
string string string

String
string



Operation schemas

The following are the schema definitions for each operation that is contained within the requested operations of a heartbeat response. It is expected that C2 agents adhere to this structure

Clear

 The clear operation uses name of connection or repositories to clear either the connections or the repositories. In the case of a connection the content contains the operation arguments, in which the value defines the connection name to clear.


Key Value
Operation heartbeat
Requested_operations

Name Operationid Operation Content
connection string clear

Unique_map_key
connection_name

Key Value
Operation heartbeat
Requested_operations

Name Operationid Operation
repositories string clear



Update

Update allows the C2 server to update either the c2 agent or provide a URI from which we download the new flow configuration through a GET request.


Key Value
Operation heartbeat
Requested_operations

Name Operationid Operation Content
configuration string update

Location
HTTP or HTTPS URL



The following activity diagram depicts the flow of updating an agent from failure to success. 

Update Agent Activity Diagram




Key Value
Operation heartbeat
Requested_operations

Name Operationid Operation Content
c2 string update

Option_name
option_value



Start
 Start starts a previously stopped command. If a start is called on a component that is already started, nothing should occur other than an acknowledgement. Name defines the component to start.


Key Value
Operation heartbeat
Requested_operations

Name Operationid Operation
component name string start



Stop
 Stop stops a component that is started. Components can be the FlowController, processors, or RPGs


Key Value
Operation heartbeat
Requested_operations

Name Operationid Operation
component name string stop


Transfer
 Transfer will download an object from the C2 designator ( server or other location ). The location URI will be provided by the update JSON. This transfer will not be a JSON
response but will instead be the binary object. The hash of the object will be the acknowledgement ID for the transfer. 


Key Value
Operation acknowledge
Operationid hash of object

Restart
  Attempts to restart the component defined within name


Key Value
Operation heartbeat
Requested_operations

Name Operationid Operation
component name string restart


Pause
  Pauses the execution of flows on the C2 agent (if the agent is running and is not in paused state), while the agent keeps running and heartbeating.


Key Value
Operation heartbeat
Requested_operations

Name Operationid Operation
component name string pause


Resume
  Resumes the execution of flows on the C2 agent if the agent is in paused state.


Key Value
Operation heartbeat
Requested_operations

Name Operationid Operation
component name string resume


Acknowledgements.

 Acknowledgements occur through a separate URL. This URL will receive a POST that contains the following payload, which acknowledges that the operation ID was received and executed.

Key Value
Operation acknowledge
Operationid string


MQTT Protocol

MQTT can be used as a connecting protocol in lieu of a RESTFul Service. Additionally, MQTT can be used within an enclave and then as conversion to RESTFul to support MQTT → HTTP comms. 

Operations and versioning

Apache NIFi MiNiFI C++ automatically supports versions 4, 3, and 2. Version one is not supported by the agent. Agents are backwards compatible and forward compatible. 

C2 role delegation

Since C2  defines operations for agents and a conversion protocol, agents can be used for role delegation. This means that Agents can act as a C2 server. This design is intentional, and supports the eventual automation of capabilities. Version four

begins implementing facilities that will be used for autonomous control. Agents can control each other and execute commands based on defined policies. Policies are controller services and configurations that enable activities to be that allow

autonomous and near-autonomous control of subnets of agents. 

C2 Policies

This sections defines the sub pages for C2 policies defined as controller services. 

C2 Failure Policies

C2 Update Policies

Operations and their operands for agents

Operation NameDescriptionoperand/namecontent/args

ACKNOWLEDGE

Operation used by MiNiFi C2 agents to acknowledge the receipt and execution of a C2 server requested operation
N/A
CLEARClear repositoriesrepositoriesN/A

CLEAR

Clears the connection queuesconnection

connection1=<connection name>, connection2=<connection 2>  ...


Will also accept a list

<connection name1>,<connection name2>, ...

CLEARClears component statecorecomponentstate

corecomponent1=<component name>, corecomponent2=<component 2>  ...

DESCRIBE

Return metricsmetricsmetricsClass=<metric class to obtain>
DESCRIBEReturn configuration optionsconfigurationN/A
DESCRIBEReturn agent manifestmanifestN/A
DESCRIBEReturn backtraces from the state monitorjstackN/A
DESCRIBEReturn all core component statescorecomponentstateN/A
HEARTBEATheartbeat operation – may contain embedded heartbeats.N/AN/A
PAUSEPauses C2 agentsN/AN/A

RESTART

Restarts C2 agentsN/AN/A
RESUMEResumes C2 agentsN/AN/A

START

Starts components within the C2 agents

C2

FlowController

<name of component to start>

N/A
STOPStops components within the C2 agent

C2

FlowController

<name of component to stop>

N/A
TRANSFERTransfers an object between the C2 agent and C2 designator.debugN/A

UPDATE

Update flowconfigurationlocation=<URL to updated flow file> or configuration_data=<flow file yaml content>
UPDATEUpdate configuration propertyproperties

propertykey1=propertyvalue1, propertykey2=propertyvalue2 ...

UPDATEDownload an assetassetfile="filename.txt", url="/c2/asset/6c8052a7-93ec-42d2-aa78-69217e3385a7", forceDownload=false


forceDownload: If true, existing files with the same name are overwritten. If false, existing files are not downloaded again.

Future Work

Future architecture of C2 should be open to the discussion of distributed architectures and multiple heads ( i.e. in a client server multiple client/servers in the case where we can talk to geographically distributed agents ). 


  • GPS based C2 server – using location information to identify and locate C2 servers.
    • C2 specific DNS responses.  could we use DNS to simply provide us the closest C2 server?
  • Multiple C2 servers providing separate keys to act as security arbiters.




  • No labels

1 Comment

  1. Marc Parisi

    Example response:

    {
    "operation": "heartbeat",
    "metrics": {
        "ProcessMetrics": {
            "CpuMetrics": {
                "involcs": "1609"
            },
            "MemoryMetrics": {
                "maxrss": "19337216"
            }
        },
        "QueueMetrics": {
            "GetFile/success/LogAttribute": {
                "datasize": "0",
                "datasizemax": "1073741824",
                "queued": "0",
                "queuedmax": "10000"
            }
        },
        "RepositoryMetrics": {
            "flowfilerepository": {
                "full": "0",
                "running": "1",
                "size": "0"
            },
            "provenancerepository": {
                "full": "0",
                "running": "1",
                "size": "0"
            }
        }
    },
    "AgentInformation": {
        "AvailableClasses": [
            {
                "properties": [
                    "Hostname Attribute",
                    "IP Attribute",
                    "Network Interface Name"
                ],
                "class_name": "org::apache::nifi::minifi::processors::AppendHostInfo",
                "supportsDynamicProperties": "false"
            },
            {
                "properties": [
                    "Max Bin Age",
                    "Maximum Group Size",
                    "Maximum Number of Entries",
                    "Maximum number of Bins",
                    "Minimum Group Size",
                    "Minimum Number of Entries"
                ],
                "class_name": "org::apache::nifi::minifi::processors::BinFiles",
                "supportsDynamicProperties": "false"
            },
            {
                "properties": [
                    "Compression Format",
                    "Compression Level",
                    "Mode",
                    "Update Filename"
                ],
                "class_name": "org::apache::nifi::minifi::processors::CompressContent",
                "supportsDynamicProperties": "false"
            },
            {
                "properties": [
                    "Batch Duration",
                    "Command",
                    "Command Arguments",
                    "Redirect Error Stream",
                    "Working Directory"
                ],
                "class_name": "org::apache::nifi::minifi::processors::ExecuteProcess",
                "supportsDynamicProperties": "false"
            },
            {
                "properties": [
                    "Module Directory",
                    "Script Body",
                    "Script Engine",
                    "Script File"
                ],
                "class_name": "org::apache::nifi::minifi::processors::ExecuteScript",
                "supportsDynamicProperties": "false"
            },
            {
                "properties": [
                    "Attribute"
                ],
                "class_name": "org::apache::nifi::minifi::processors::ExtractText",
                "supportsDynamicProperties": "false"
            },
            {
                "properties": [
                    "Path"
                ],
                "class_name": "org::apache::nifi::minifi::processors::FocusArchiveEntry",
                "supportsDynamicProperties": "false"
            },
            {
                "properties": [
                    "Batch Size",
                    "Data Format",
                    "File Size",
                    "Unique FlowFiles"
                ],
                "class_name": "org::apache::nifi::minifi::processors::GenerateFlowFile",
                "supportsDynamicProperties": "false"
            },
            {
                "properties": [
                    "Batch Size",
                    "File Filter",
                    "Ignore Hidden Files",
                    "Input Directory",
                    "Keep Source File",
                    "Maximum File Age",
                    "Maximum File Size",
                    "Minimum File Age",
                    "Minimum File Size",
                    "Polling Interval",
                    "Recurse Subdirectories"
                ],
                "class_name": "org::apache::nifi::minifi::processors::GetFile",
                "supportsDynamicProperties": "false"
            },
            {
                "properties": [
                    "SSL Context Service",
                    "Stay Connected",
                    "concurrent-handler-count",
                    "connection-attempt-timeout",
                    "end-of-message-byte",
                    "endpoint-list",
                    "receive-buffer-size"
                ],
                "class_name": "org::apache::nifi::minifi::processors::GetTCP",
                "supportsDynamicProperties": "false"
            },
            {
                "properties": [
                    "Always Output Response",
                    "Attributes to Send",
                    "Connection Timeout",
                    "Content-type",
                    "Disable Peer Verification",
                    "HTTP Method",
                    "Include Date Header",
                    "Proxy Host",
                    "Proxy Port",
                    "Read Timeout",
                    "Remote URL",
                    "SSL Context Service",
                    "Use Chunked Encoding",
                    "invokehttp-proxy-password",
                    "invokehttp-proxy-user",
                    "send-message-body"
                ],
                "class_name": "org::apache::nifi::minifi::processors::InvokeHTTP",
                "supportsDynamicProperties": "false"
            },
            {
                "properties": [
                    "Battery Capacity Path",
                    "Battery Status Path",
                    "Low Battery Threshold",
                    "Trigger Threshold",
                    "Wait Period"
                ],
                "class_name": "org::apache::nifi::minifi::controllers::LinuxPowerManagerService",
                "supportsDynamicProperties": "false"
            },
            {
                "properties": [
                    "Authorized DN Pattern",
                    "Base Path",
                    "HTTP Headers to receive as Attributes (Regex)",
                    "Listening Port",
                    "SSL Certificate",
                    "SSL Certificate Authority",
                    "SSL Minimum Version",
                    "SSL Verify Peer"
                ],
                "class_name": "org::apache::nifi::minifi::processors::ListenHTTP",
                "supportsDynamicProperties": "false"
            },
            {
                "properties": [
                    "Max Batch Size",
                    "Max Number of TCP Connections",
                    "Max Size of Socket Buffer",
                    "Message Delimiter",
                    "Parse Messages",
                    "Port",
                    "Protocol",
                    "Receive Buffer Size"
                ],
                "class_name": "org::apache::nifi::minifi::processors::ListenSyslog",
                "supportsDynamicProperties": "false"
            },
            {
                "properties": [
                    "Attributes to Ignore",
                    "Attributes to Log",
                    "Log Level",
                    "Log Payload",
                    "Log prefix"
                ],
                "class_name": "org::apache::nifi::minifi::processors::LogAttribute",
                "supportsDynamicProperties": "false"
            },
            {
                "properties": [
                    "After",
                    "Before",
                    "Destination",
                    "Operation",
                    "Target"
                ],
                "class_name": "org::apache::nifi::minifi::processors::ManipulateArchive",
                "supportsDynamicProperties": "false"
            },
            {
                "properties": [
                    "Correlation Attribute Name",
                    "Delimiter Strategy",
                    "Demarcator File",
                    "Footer File",
                    "Header File",
                    "Keep Path",
                    "Max Bin Age",
                    "Maximum Group Size",
                    "Maximum Number of Entries",
                    "Maximum number of Bins",
                    "Merge Format",
                    "Merge Strategy",
                    "Minimum Group Size",
                    "Minimum Number of Entries"
                ],
                "class_name": "org::apache::nifi::minifi::processors::MergeContent",
                "supportsDynamicProperties": "false"
            },
            {
                "properties": [
                    "Conflict Resolution Strategy",
                    "Create Missing Directories",
                    "Directory",
                    "Maximum File Count"
                ],
                "class_name": "org::apache::nifi::minifi::processors::PutFile",
                "supportsDynamicProperties": "false"
            },
            {
                "properties": [
                    "CA Certificate",
                    "Client Certificate",
                    "Passphrase",
                    "Private Key"
                ],
                "class_name": "org::apache::nifi::minifi::controllers::SSLContextService",
                "supportsDynamicProperties": "false"
            },
            {
                "properties": [
                    "File to Tail",
                    "Input Delimiter",
                    "State File"
                ],
                "class_name": "org::apache::nifi::minifi::processors::TailFile",
                "supportsDynamicProperties": "false"
            },
            {
                "class_name": "org::apache::nifi::minifi::processors::UnfocusArchiveEntry",
                "supportsDynamicProperties": "false"
            },
            {
                "class_name": "org::apache::nifi::minifi::processors::UpdateAttribute",
                "supportsDynamicProperties": "true"
            }
        ],
        "BuildInformation": {
            "build_date": "Tue Mar  6 09:20:53 EST 2018",
            "build_rev": "9b01c508e53c3108d4f9273b2751c7311648aece",
            "build_version": "0.5.0",
            "compiler": "/Library/Developer/CommandLineTools/usr/bin/c++",
            "compiler_flags": "  -std=c++11 -Wall",
            "device_id": "classone"
        },
        "Extensions": [
            "minifi-archive-extensions",
            "minifi-civet-extensions",
            "minifi-expression-language-extensions",
            "minifi-http-curl",
            "minifi-rocksdb-repos",
            "minifi-script-extensions"
        ],
        "NetworkInfo": {
            "deviceid": "2166229772469413744",
            "hostname": "MachineName",
            "ip": "IP"
        },
        "SystemInformation": {
            "machinearch": "x86_64",
            "physicalmem": "17179869184",
            "vcores": "8"
        }
    },
    "Components": {
        "FlowController": "enabled",
        "GetFile": "enabled",
        "LogAttribute": "enabled"
    },
    "state": {
        "running": "true",
        "uptime": "2889"
    }
}