Sqoop 2 Actors
|Sqoop Admins||Creating LINK objects and LINK Config Inputs|
|Sqoop Users||Creating and Executing Sqoop JOBs|
|Sqoop Connector developers||Developing Connectors and Connector related config semantics|
Sqoop 2 Entities
Since 1.99.4 release we renamed a sqoop entities and added one new entity "CONFIGURABLE" that acts as a one of the core entities representing sqoop object exposing configs. Here is the list of configs
|Sqoop Entity||Before 1.99.4 It was called?||Sqoop Java Classes||Relationships to to other entities||Description||CRUD Operations supported via command shell or REST|
Top Level Entity
Represents a core entity that exposes config objects and used in sqoop job lifecycle.
Configurable have a associated version that acts as a identifier for connector config upgrades.
is a type of configurable
There can be many connectors registered to the sqoop server
Connectors and their exposed config objects are registered to the sqoop server at run-time when server starts. They are actual code artifacts packaged as jars. But they are also stored in the sqoop persistent store referred to as the repository to uniquely identify them and their config objects they expose.
Connector upgrades are also supported across sqoop releases.
is a type of configurable
There is only one Driver object representing sqoop in the system
Driver is also registered to the sqoop server during server start time along with its associated config objects.
It also has a upgrade path similar to connectors.
|CONFIG||FORM||Top Level Entity|
MConfigType with supported config types are
READ ONLY, created once during the server start up, we do not allow update/delete via shell or REST
Note: We do not yet allow creating/deleting/editing configs at runtime via shell/ REST, and we will not probably do that ever since we want the config objects be declared in code via the @Config annotation. But Config and Inputs objects can be deleted as part of the configurable upgrade code path. Thus connector developers can delete/update it but not the sqoop users
|INPUT ( Keys and Values )||Same|
Concrete classes for each supported types
Represents the key-value pairs for a given config.
READ ONLY for Input Keys
Input keys are created as configs are registered. We do not allow deletes/updates via the shell/REST.
RU for Input values
Input values can be edited per config object
See SQOOP-1516 for rest apis related to config input Read/Updates per job/configId
Represents the config inputs required to physically connect to the data-source a connector represents. Hence it is associated with a connector.
It has mainly one config object represented by MLinkConfig
Represents the sqoop job. It encapsulates all the required configs to run the sqoop job.
Primarily the sqoop job has the 3 main components, the FROM, TO and the DRIVER.
FROM and its related MFromConfig represent the config-inputs-values required to Extract data from the source
TO and its related MToConfig represent the config-inputs-values required to load data to the destination
DRIVER and its related MDriverConfig the config-inputs-values required by the execution engine that runs the sqoop job optimally.
Represents the job run details. Includes the job status, job counters and metrics from the job execution engine
Rest API changes :
SQOOP-1516 ( scheduled for 1.99.5 though )