This wiki documents the steps to get started with Gobblin-as-a-Service and single tenant Azkaban Orchestrator.
Setup
Download code (master)
wget https://github.com/apache/incubator-gobblin
Build distribution
cd incubator-gobblin; ./gradlew clean build
Locate and extract the distribution
cd build/gobblin-distribution/distributions; tar xvf gobblin-distribution-*.tar.gz
Move into the extracted directory and start the service after modifying configuration (as described in next section)
cd gobblin-dist; bin/gobblin-service.sh start
Configuration
Topology Catalog
To enable Gobblin-as-a-Service to talk to an Azkaban deployment, you need to modify the application.conf
vi conf/service/application.conf
Config to specify an Azkaban deployment looks like:
topologySpecFactory.topologyNames=<azkaban-name> topologySpecFactory.<azkaban-name>.gobblin.service.azkaban.username=<username> topologySpecFactory.<azkaban-name>.gobblin.service.azkaban.password=<password> topologySpecFactory.<azkaban-name>.gobblin.service.azkaban.server.url="<url>" topologySpecFactory.<azkaban-name>.description="AzkabanTopology" topologySpecFactory.<azkaban-name>.version="1" topologySpecFactory.<azkaban-name>.uri="GobblinAzkaban" topologySpecFactory.<azkaban-name>.specExecutorInstanceProducer.class="org.apache.gobblin.service.modules.orchestration.AzkabanSpecExecutorInstanceProducer" topologySpecFactory.<azkaban-name>.specExecInstance.capabilities="any:any"
Flow Catalog
No change is required for Flow Catalog or its setup
Template Catalog
By default Template Catalog is set to use directory: /tmp/templateCatalog
To modify the default template catalog location, change config in conf/service/application.conf
vi conf/service/application.conf
And change the following value to your desired directory:
gobblin.service.templateCatalogs.fullyQualifiedPath="file:///tmp/templateCatalog"
Adding Template
You can add new templates for the Gobblin-as-a-Service to use by adding them in the template catalog. An example template is below:
# Azkaban specific properties type=java # Mandatory input properties (to be supplied via REST API call property bag) gobblin.template.required_attributes="titles" # Pre-defined properties job.name=PullFromWikipedia job.group=Wikipedia job.description=A getting started example for Gobblin source.class=org.apache.gobblin.example.wikipedia.WikipediaSource source.page.titles=${titles} wikipedia.api.rooturl="https://en.wikipedia.org/w/api.php" wikipedia.avro.schema="{\"namespace\": \"example.wikipedia.avro\",\"type\": \"record\",\"name\": \"WikipediaArticle\",\"fields\": [{\"name\": \"revid\", \"type\": [\"double\", \"null\"]},{\"name\": \"pageid\", \"type\": [\"double\", \"null\"]},{\"name\": \"title\", \"type\": [\"string\", \"null\"]},{\"name\": \"user\", \"type\": [\"string\", \"null\"]},{\"name\": \"anon\", \"type\": [\"string\", \"null\"]},{\"name\": \"userid\", \"type\": [\"double\", \"null\"]},{\"name\": \"timestamp\", \"type\": [\"string\", \"null\"]},{\"name\": \"size\", \"type\": [\"double\", \"null\"]},{\"name\": \"contentformat\", \"type\": [\"string\", \"null\"]},{\"name\": \"contentmodel\", \"type\": [\"string\", \"null\"]},{\"name\": \"content\", \"type\": [\"string\", \"null\"]}]}" gobblin.wikipediaSource.maxRevisionsPerPage=10 extract.namespace=org.apache.gobblin.example.wikipedia writer.builder.class=org.apache.gobblin.writer.ConsoleWriterBuilder data.publisher.type=org.apache.gobblin.publisher.NoopPublisher
Execution
Creating a new Flow
You can create a new Flow using the above template through the following curl command:
curl http://localhost:<port>/flowconfigs -X POST -H 'X-RestLi-Method: create' -H 'X-RestLi-Protocol-Version: 2.0.0' --data '{"id": {"flowName":"<flow name>", "flowGroup":"<flow group>"},"schedule":{"cronSchedule":"<cron schedule>", "runImmediately": true}, "templateUris" : "FS:///wikipedia.template", "properties" : {"gobblin.flow.sourceIdentifier" : "any", "gobblin.flow.destinationIdentifier" : "any", "titles" : "Apache"}}'
Note
- Notice that we have specified "titles" as a property in the curli command (as required by the template)
- You should see an Azkaban project created when your flow executes