Top Level Entity

Represents a core entity that exposes config objects and used in sqoop job lifecycle.

Configurable have a associated version that acts as a identifier for connector config upgrades.

/**
 * Represents the sqoop entities that can own configs
 */
public enum MConfigurableType {
  /** Connector as a owner of config keys */
  CONNECTOR,
  /** Driver as a owner of config keys */
  DRIVER;
}

READ ONLY.

• /v1/configurable/driver
• /v1/configurable/connector/[cid]

• HAS 1-n CONFIG objects
• HAS 1-n LINK objects

is a type of configurable

There can be many connectors registered to the sqoop server

READ ONLY

Connectors and their exposed config objects are registered to the sqoop server at run-time when server starts. They are actual code artifacts packaged as jars. But they are also stored in the sqoop persistent store referred to as the repository to uniquely identify them and their config objects they expose.

Connector upgrades are also supported across sqoop releases.

• /v1/connectors - [GET] Get all Connectors
• /v1/connector/[cname] or /v1/connector/[cid] - [GET] - Get Connector

• HAS 1-n CONFIG objects

is a type of configurable

There is only one Driver object representing sqoop in the system

READ ONLY.

Driver is also registered to the sqoop server during server start time along with its associated config objects.

It also has a upgrade path similar to connectors.

• /v1/driver - [GET]- Get Sqoop Driver

MConfigType with supported config types are

public enum MConfigType {
  /** Unknown config type */
  OTHER,
  @Deprecated
  // NOTE: only exists to support the connector data upgrade path
  CONNECTION,
  /** link config type */ 
  LINK,
  /** Job config type */
  JOB;
}

RU, created once during the server start up, we do not allow delete via shell or REST

We do not yet allow creating configs at runtime and we will not probably do that since we want the config objects be declared in code via the @Config annotation

Note: Config and Inputs can be deleted as part of the configurable upgrade code path. Thus connector developers can delete it.

See SQOOP-1516 for rest apis related to config read/update.

MInput.java an abstract class and @Input annotation

Concrete classes for each supported types

MIntegerInput.java

MStringInput.java

• Associated with a CONFIG object

Represents the key-value pairs for a given config.

MInputTypes supported are

public enum MInputType {
  /** Unknown input type */
  OTHER,
  /** String input type */
  STRING,
  /** Map input type */
  MAP,
  /** Integer input type */
  INTEGER,
  /** Boolean input type */
  BOOLEAN,
  /** String based input that can contain only predefined values **/
  ENUM,
  ;
}

same as above, input values can be read and edited.

MLink.java

• Associated with a CONNECTOR
• HAS a CONFIG-INPUT object

Represents the config inputs required to physically connect to the data-source a connector represents. Hence it is associated with a connector.

It has mainly one config object represented by MLinkConfig

CRUD

• /v1/links/ - [GET] Get all links
• /v1/links?cname=[cname] - [GET] Get all links by Connector
• /v1/link/[lname] or /v1/link/[lid] - [GET] - Get Link
• /v1/link - [POST] - Create Link
• /v1/link/[lname] or /v1/link/[lid] - [PUT] - Update Link
• /v1/link/[lname] or /v1/link/[lid] - [DELETE] - Delete Link
• /v1/link/[lid]/enable or /v1/link/[lname]/enable - [PUT] - Enable Link
• /v1/link/[lid]/disable - [PUT] - Disable Link

MJob.java

MFromConfig.java

MToConfig.java

MDriverConfig.java

• HAS 3 CONFIG-INPUT objects
• HAS 1-n SUBMISSIONS

Represents the sqoop job. It encapsulates all the required configs to run the sqoop job.

Primarily the sqoop job has the 3 main components, the FROM, TO and the DRIVER.

FROM and its related MFromConfig represent the config-inputs-values required to Extract data from the source

TO and its related MToConfig represent the config-inputs-values required to load data to the destination

DRIVER and its related MDriverConfig the config-inputs-values required by the execution engine that runs the sqoop job optimally.

CRUD

• /v1/job - [POST] - Create Job
• /v1/job/[jid] - [PUT] - Update Job
• /v1/job/[jid] - [DELETE] - Delete Job
• /v1/job/[jid]/enable - [PUT] - Enable Job
• /v1/job/[jid]/disable - [PUT] - Disable Job
• /v1/job/[jid]/start or /v1/job/[jname]/start - [PUT]- Start Job
• /v1/job/[jid]/stop or /v1/job/[jname]/stop - [PUT]- Stop Job
• /v1/job/[jid]/status or /v1/job/[jname]/status - [GET]- Get Job Status

Represents the job run details. Includes the job status, job counters and metrics from the job execution engine

READ ONLY

• /v1/submissions? - [GET] - Get all job Submissions
• /v1/submissions?jname=[jname] - [GET] - Get Submissions by Job