Top Level Entity

Represents a core entity that exposes config objects and used in sqoop job lifecycle.

Configurable have a associated version that acts as a identifier for connector config upgrades.

/**
 * Represents the sqoop entities that can own configs
 */
public enum MConfigurableType {
  /** Connector as a owner of config keys */
  CONNECTOR,
  /** Driver as a owner of config keys */
  DRIVER;
}

READ ONLY.

• /v1/configurable/driver
• /v1/configurable/connector/[cid]

• HAS 1-n CONFIG objects
• HAS 1-n LINK objects

is a type of configurable

There can be many connectors registered to the sqoop server

READ ONLY

Connectors and their exposed config objects are registered to the sqoop server at run-time when server starts. They are actual code artifacts packaged as jars. But they are also stored in the sqoop persistent store referred to as the repository to uniquely identify them and their config objects they expose.

Connector upgrades are also supported across sqoop releases.

• /v1/connectors - [GET] Get all Connectors
• /v1/connector/[cname] or /v1/connector/[cid] - [GET] - Get Connector

• HAS 1-n CONFIG objects

is a type of configurable

There is only one Driver object representing sqoop in the system

READ ONLY.

Driver is also registered to the sqoop server during server start time along with its associated config objects.

It also has a upgrade path similar to connectors.

• /v1/driver - [GET]- Get Sqoop Driver

MConfigType with supported config types are

public enum MConfigType {
  /** Unknown config type */
  OTHER,
  @Deprecated
  // NOTE: only exists to support the connector data upgrade path
  CONNECTION,
  /** link config type */ 
  LINK,
  /** Job config type */
  JOB;
}

READ ONLY, created once during the server start up, we do not allow update/delete via shell or REST

Note: We do not yet allow creating/deleting/editing configs at runtime via shell/ REST, and we will not probably do that ever since we want the config objects be declared in code via the @Config annotation. But Config and Inputs objects can be deleted as part of the configurable upgrade code path. Thus connector developers can delete/update it but not the sqoop users

MInput.java an abstract class and @Input annotation

Concrete classes for each supported types

MIntegerInput.java

MStringInput.java

• Associated with a CONFIG object

Represents the key-value pairs for a given config.

MInputTypes supported are

public enum MInputType {
  /** Unknown input type */
  OTHER,
  /** String input type */
  STRING,
  /** Map input type */
  MAP,
  /** Integer input type */
  INTEGER,
  /** Boolean input type */
  BOOLEAN,
  /** String based input that can contain only predefined values **/
  ENUM,
  ;
}

READ ONLY for Input Keys

Input keys are created as configs are registered. We do not allow deletes/updates via the shell/REST.

RU for Input values

Input values can be edited per config object

See SQOOP-1516 for rest apis related to config input Read/Updates per job/configId

MLink.java

• Associated with a CONNECTOR
• HAS a CONFIG-INPUT object

Represents the config inputs required to physically connect to the data-source a connector represents. Hence it is associated with a connector.

It has mainly one config object represented by MLinkConfig

CRUD

• /v1/links/ - [GET] Get all links
• /v1/links?cname=[cname] - [GET] Get all links by Connector
• /v1/link/[lname] or /v1/link/[lid] - [GET] - Get Link
• /v1/link - [POST] - Create Link
• /v1/link/[lname] or /v1/link/[lid] - [PUT] - Update Link
• /v1/link/[lname] or /v1/link/[lid] - [DELETE] - Delete Link
• /v1/link/[lid]/enable or /v1/link/[lname]/enable - [PUT] - Enable Link
• /v1/link/[lid]/disable - [PUT] - Disable Link

MJob.java

MFromConfig.java

MToConfig.java

MDriverConfig.java

• HAS 3 CONFIG-INPUT objects
• HAS 1-n SUBMISSIONS

Represents the sqoop job. It encapsulates all the required configs to run the sqoop job.

Primarily the sqoop job has the 3 main components, the FROM, TO and the DRIVER.

FROM and its related MFromConfig represent the config-inputs-values required to Extract data from the source

TO and its related MToConfig represent the config-inputs-values required to load data to the destination

DRIVER and its related MDriverConfig the config-inputs-values required by the execution engine that runs the sqoop job optimally.

CRUD

• /v1/job - [POST] - Create Job
• /v1/job/[jid] - [PUT] - Update Job
• /v1/job/[jid] - [DELETE] - Delete Job
• /v1/job/[jid]/enable - [PUT] - Enable Job
• /v1/job/[jid]/disable - [PUT] - Disable Job
• /v1/job/[jid]/start or /v1/job/[jname]/start - [PUT]- Start Job
• /v1/job/[jid]/stop or /v1/job/[jname]/stop - [PUT]- Stop Job
• /v1/job/[jid]/status or /v1/job/[jname]/status - [GET]- Get Job Status

Represents the job run details. Includes the job status, job counters and metrics from the job execution engine

READ ONLY

• /v1/submissions? - [GET] - Get all job Submissions
• /v1/submissions?jname=[jname] - [GET] - Get Submissions by Job