Ambari Blueprints are a declarative definition of a cluster. With a Blueprint, you specify a Stack, the Component layout and the Configurations to materialize a Hadoop cluster instance (via a REST API) without having to use the Ambari Cluster Install Wizard.
JIRA | Description |
---|---|
AMBARI-4467 | Blueprints REST resource. |
AMBARI-5077 | Provision cluster with blueprint. |
AMBARI-4786 | Export blueprints from running/existing cluster. |
AMBARI-5114 | Configurations with blueprints. |
AMBARI-6275 | Add hosts using blueprints. |
AMBARI-10750 | 2.1 blueprint changes. |
The following table lists the basic Blueprints API resources.
Resource | Description |
---|---|
GET /blueprints | Returns the available blueprints. |
POST /blueprints/:name | Creates a blueprint. |
POST /clusters/:name | Creates a cluster. |
GET /clusters/:name?format=blueprint | Export the current cluster layout as a blueprint. |
The API calls on this wiki page include the HTTP Method (for example:
|
Install the Ambari Server, run setup and start. Install the Ambari Agents on all hosts and perform manual registration.
A blueprint can be created by hand or can be created by exporting a blueprint from an existing cluster.
To export a cluster from an existing cluster: |
POST /api/v1/blueprints/:blueprintName
Request body is blueprint created in Step 1.
To disable topology validation and register a blueprint:
POST /api/v1/blueprints/:blueprintName?validate_topology=false
Disabling topology validation allows a user to force registration of a blueprint that fails topology validation.
Map Physical Hosts to Blueprint: Create the mapping between blueprint host groups and physical hosts.
Provide Cluster Specific Configuration Overrides : Configuration can be applied at the cluster and host group scope and overrides any configurations specified in the blueprint.
There are scenarios where public Stack repositories may not be accessible during creation of cluster via blueprint or an alternate repository is required for Stack.
To use a local or alternate repository:
PUT /api/v1/stacks/:stack/versions/:stackVersion/operating_systems/:osType/repositories/:repoId { "Repositories” : { "base_url” : ”<CUSTOM_REPO_BASE_URL>", "verify_base_url” : true } } |
This API may be invoked multiple times to set Base URL for multiple OS types or Stack versions. If this step is not performed, by default, blueprints will use the latest Base URL defined in the Stack.
POST /api/v1/clusters/:clusterName
Request body includes blueprint name, host mappings and configurations from Step 3.
Request is asynchronous and returns a /requests
URL which can be used to monitor progress.
Using the /requests
URL returned in Step 4, monitor the progress of the tasks associated with cluster creation.
Ambari Blueprints currently does not support creating cluster reflecting a High Availability topology. tracks these enhancements.
Ambari 2.0 adds support for deploying High Availability clusters in Blueprints. Please see Blueprint Support for HA Clusters for more information on this topic. |
yum install ambari-server
ambari-server setup
ambari-server start
yum install ambari-agent
vi /etc/ambari-agent/conf/ambari-agent.ini
hostname=c6401.ambari.apache.org
ambari-agent start
http://your.ambari.server:8080/api/v1/hosts
A blueprint document is in JSON format and has the following structure:
{ "configurations" : [ { "configuration-type" : { "property-name" : "property-value", "property-name2" : "property-value" } }, { "configuration-type2" : { "property-name" : "property-value" } } ... ], "host_groups" : [ { "name" : "host-group-name", "components" : [ { "name" : "component-name" }, { "name" : "component-name2", "provision_action" : "(INSTALL_AND_START | INSTALL_ONLY)" } ... ], "configurations" : [ { "configuration-type" : { "property-name" : "property-value" } } ... ], "cardinality" : "1" } ], "settings" : [ "deployment_settings": [ {"skip_failure":"true"} ], "repository_settings":[ { "override_strategy":"ALWAYS_APPLY", "operating_system":"redhat7", "repo_id":"HDP", "base_url":"http://myserver/hdp" }, { "override_strategy":"APPLY_WHEN_MISSING", "operating_system":"redhat7", "repo_id":"HDP-UTIL-1.1", "base_url":"http://myserver/hdp-util" } ], "recovery_settings":[ {"recovery_enabled":"true"} ], "service_settings":[ { "name":"SERVICE_ONE", "recovery_enabled":"true" }, { "name":"SERVICE_TWO", "recovery_enabled":"true" } ], "component_settings":[ { "name":"COMPONENT_A_OF_SERVICE_ONE" "recover_enabled":"true" }, { "name":"COMPONENT_B_OF_SERVICE_TWO", "recover_enabled":"true" } ] ], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.1", "security" : { "type" : "(NONE | KERBEROS)", "kerberos_descriptor" : { ... } } } } |
settings: An optional section to provide additional configuration for cluster behavior during and after the blueprint deployment. You can provide configurations for the following properties:
recovery_settings: A section to specify if all services (globally) should be set with auto restart once the cluster is deployed. To configure it, specify "recover_enabled" property to either "true" (auto restart), or "false" (do not auto restart).
For example:
"settings": [
"recovery_settings":[
{
"recovery_enabled":"true"
}
]
],
repository_settings: A section to specify custom repository URLs for the blueprint deployment. This section allows you to provide custom URLs to override the default ones. Without this section, you will need to update the repository URLs via REST API before deploying the cluster with the blueprint. "override_strategy" can be "ALWAYS_APPLY" ( always override the default one), or "APPLY_WHEN_MISSING" (only add it if no repository exists with the specific operating system and the repository id information). Repository URLs stored in the Ambari server database will be used if the blueprint does not have the "repository_settings" section.
For example:
settings: [
"repository_settings":[
{
"override_strategy":"ALWAYS_APPLY"
"operating_system":"redhat7",
"repo_id":"HDP",
"base_url":"http://myserver/hdp"
},
{
"override_strategy":"APPLY_WHEN_MISSING",
"operating_system":"redhat7",
"repo_id":"HDP-UTIL-1.1",
"base_url":"http://myserver/hdp-util"
}
]
]
A Cluster Creation Template is in JSON format and has the following structure:
{ "blueprint" : "blueprint-name", "default_password" : "super-secret-password", "provision_action" : "(INSTALL_AND_START | INSTALL_ONLY)" "configurations" : [ { "configuration-type" : { "property-name" : "property-value" } } ... ], "host_groups" :[ { "name" : "host-group-name", "configurations" : [ { "configuration-type" : { "property-name" : "property-value" } } ], "hosts" : [ { "fqdn" : "host.domain.com" }, { "fqdn" : "host2.domain.com" } ... ] } ... ], "credentials" : [ { "alias" : "kdc.admin.credential", "principal" : "{PRINCIPAL}", "key" : "{KEY}", "type" : "(TEMPORARY | PERSISTED)" } ], "security" : { "type" : "(NONE | KERBEROS)", "kerberos_descriptor" : { ... } } } |
Starting in Ambari version 2.1.0, it is possible to specify a host count and a host predicate in the cluster creation template host group section instead of a host name.
|
Starting in Ambari version 2.2.0, it is possible to specify configuration recommendation strategy in the cluster creation template.
|
Starting in Ambari version 2.2.1, it is possible to specify the host rack info in the cluster creation template (AMBARI-14600).
|
Cluster Creation Template Structure: Host Mappings and Configuration Field Descriptions
{ "host_groups" : [ { "name" : "host_group_1", "components" : [ { "name" : "NAMENODE" }, { "name" : "SECONDARY_NAMENODE" }, { "name" : "DATANODE" }, { "name" : "HDFS_CLIENT" }, { "name" : "RESOURCEMANAGER" }, { "name" : "NODEMANAGER" }, { "name" : "YARN_CLIENT" }, { "name" : "HISTORYSERVER" }, { "name" : "MAPREDUCE2_CLIENT" }, { "name" : "ZOOKEEPER_SERVER" }, { "name" : "ZOOKEEPER_CLIENT" } ], "cardinality" : "1" } ], "Blueprints" : { "blueprint_name" : "single-node-hdfs-yarn", "stack_name" : "HDP", "stack_version" : "2.4" } } |
Post the blueprint to the "single-node-hdfs-yarn" resource to the Ambari Server.
POST /api/v1/blueprints/single-node-hdfs-yarn ... [ Request Body is the example blueprint defined above ] ... 201 - Created |
We are performing a single-node install and the blueprint above has one host group. Therefore, for our cluster instance, we define one host in host_group_1 and reference the single-node-hdfs-yarn blueprint.
Explicit Host Name Example
{ "blueprint" : "single-node-hdfs-yarn", "host_groups" :[ { "name" : "host_group_1", "hosts" : [ { "fqdn" : "c6401.ambari.apache.org" } ] } ] } |
Create Cluster Instance
Post the cluster to the Ambari Server to provision the cluster.
POST /api/v1/clusters/MySingleNodeCluster ... [ Request Body is above Cluster Creation Template ] ... 202 - Accepted { "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/MyCluster/requests/1", "Requests" : { "id" : 1, "status" : "InProgress" } } |
The blueprint ("multi-node-hdfs-yarn") below defines with two host groups (a "master" and the "slaves") which hosts the various Service components (masters, slaves and clients).
{ "host_groups" : [ { "name" : "master", "components" : [ { "name" : "NAMENODE" }, { "name" : "SECONDARY_NAMENODE" }, { "name" : "RESOURCEMANAGER" }, { "name" : "HISTORYSERVER" }, { "name" : "ZOOKEEPER_SERVER" } ], "cardinality" : "1" }, { "name" : "slaves", "components" : [ { "name" : "DATANODE" }, { "name" : "HDFS_CLIENT" }, { "name" : "NODEMANAGER" }, { "name" : "YARN_CLIENT" }, { "name" : "MAPREDUCE2_CLIENT" }, { "name" : "ZOOKEEPER_CLIENT" } ], "cardinality" : "1+" } ], "Blueprints" : { "blueprint_name" : "multi-node-hdfs-yarn", "stack_name" : "HDP", "stack_version" : "2.4" } } |
Post the blueprint to the "single-node-hdfs-yarn" resource to the Ambari Server.
POST /api/v1/blueprints/multi-node-hdfs-yarn ... [ Request Body is the example blueprint defined above ] ... 201 - Created |
We are performing a multi-node install and the blueprint above has two host groups. Therefore, for our cluster instance, we define one host in masters, two hosts in slaves and reference the multi-node-hdfs-yarn blueprint.
The below multi-node cluster creation template example uses the "host_count" and "host_predicate" syntax for the "slave" host group which is available as of Ambari 2.1.0. For older versions of Ambari, the "hosts/fqdn" syntax must be used. |
{ "blueprint" : "multi-node-hdfs-yarn", "default_password" : "my-super-secret-password", "host_groups" :[ { "name" : "master", "hosts" : [ { "fqdn" : "c6401.ambari.apache.org" } ] }, { "name" : "slaves", "host_count" : 5, "host_predicate" : "Hosts/os_type=centos6&Hosts/cpu_count=2" } ] } |
Post the cluster to the Ambari Server to provision the cluster.
POST /api/v1/clusters/MyThreeNodeCluster ... [ Request Body is above Cluster Creation Template ] ... 202 - Accepted { "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/MyThreeNodeCluster/requests/1", "Requests" : { "id" : 1, "status" : "InProgress" } } |
After creating a cluster using the Ambari Blueprint API, you may scale up the cluster using the API.
There are two forms of the API, one for adding a single host and another for adding multiple hosts.
The blueprint add hosts API is available as of Ambari 2.0. Currently, only clusters originally provisioned via the blueprint API may be scaled using this API. |
Host is specified in URL
{ "blueprint" : "multi-node-hdfs-yarn", "host_group" : "slaves" } |
Host is specified in request body
[ { "blueprint" : "multi-node-hdfs-yarn", "host_group" : "slaves", "host_name" : "c6403.ambari.apache.org" }, { "blueprint" : "multi-node-hdfs-yarn", "host_group" : "slaves", "host_name" : "c6404.ambari.apache.org" } ] |
Starting with Ambari 2.1, the fields "host_count" and "host_predicate" can also be used when adding a host.
These fields behave exactly the same as they do when specified in the cluster creation template.
[ { "blueprint" : "multi-node-hdfs-yarn", "host_group" : "slaves", "host_count" : 5, "host_predicate" : "Hosts/os_type=centos6&Hosts/cpu_count=2" } ] |
POST /api/v1/clusters/myExistingCluster/hosts/c6403.ambari.apache.org ... [ Request Body is above Single Host Add Host Template ] ... 202 - Accepted { "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/myExistingCluster/requests/1", "Requests" : { "id" : 1, "status" : "Pending" } } |
POST /api/v1/clusters/myExistingCluster/hosts ... [ Request Body is above Multiple Host Add Host Template ] ... 202 - Accepted { "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/myExistingCluster/requests/1", "Requests" : { "id" : 1, "status" : "Pending" } } |
The blueprint below could be used to setup a cluster containing three host groups with KERBEROS security. Overriding default kerberos descriptor is not necessary however specifying a few Kerberos specific properties in kerberos-env and krb5-conf is a must to setup services to use Kerberos. Note: prior to Ambari 2.4.0 use "kdc_host" instead of "kdc_hosts".
{ "configurations" : [ { "kerberos-env": { "properties_attributes" : { }, "properties" : { "realm" : "AMBARI.APACHE.ORG", "kdc_type" : "mit-kdc", "kdc_hosts" : "(kerberos_server_name)", "admin_server_host" : "(kerberos_server_name)" } } }, { "krb5-conf": { "properties_attributes" : { }, "properties" : { "domains" : "AMBARI.APACHE.ORG", "manage_krb5_conf" : "true" } } } ], "host_groups" : [ { "name" : "host_group_1", "configurations" : [ ], "components" : [ { "name" : "ZOOKEEPER_CLIENT" }, { "name" : "ZOOKEEPER_SERVER" }, { "name" : "NAMENODE" }, { "name" : "HDFS_CLIENT" }, { "name" : "DATANODE" } ], "cardinality" : "1" }, { "name" : "host_group_2", "configurations" : [ ], "components" : [ { "name" : "ZOOKEEPER_SERVER" }, { "name" : "KERBEROS_CLIENT" }, { "name" : "SECONDARY_NAMENODE" }, { "name" : "DATANODE" } ], "cardinality" : "1" }, { "name" : "host_group_3", "configurations" : [ ], "components" : [ { "name" : "ZOOKEEPER_CLIENT" }, { "name" : "ZOOKEEPER_SERVER" }, { "name" : "KERBEROS_CLIENT" }, { "name" : "HDFS_CLIENT" }, { "name" : "DATANODE" } ], "cardinality" : "1" } ], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.3", "security" : { "type" : "KERBEROS" } } } |
The Cluster Creation Template below could be used to setup a cluster containing hosts with KERBEROS security using the Blueprint from above. Overriding default kerberos descriptor is not necessary however specifying kdc.admin credentials is a must.
{ "blueprint": "kerberosBlueprint", "default_password": "admin", "host_groups": [ { "hosts": [ { "fqdn": "ambari-agent-1" } ], "name": "host_group_1", "configurations" : [ ] }, { "hosts": [ { "fqdn": "ambari-agent-2" } ], "name": "host_group_2", "configurations" : [ ] }, { "hosts": [ { "fqdn": "ambari-agent-3" } ], "name": "host_group_3", "configurations" : [ ] } ], "credentials" : [ { "alias" : "kdc.admin.credential", "principal" : "admin/admin", "key" : "admin", "type" : "TEMPORARY" } ], "security" : { "type" : "KERBEROS" }, "Clusters" : {"cluster_name":"kerberosCluster"} } |
Support for deploying HA clusters for HDFS, Yarn, and HBase has been added in Ambari 2.0. Please see the following link for more information:
Blueprint Support for HA Clusters