I’m working on scripts that fully automate the installation of IOP and BigInsights value adds.
I understand that’s an IBM product, but it shares a common code as it relates to IOP and Ambari.
I had the installation of IOP and BI value adds fully scripted for 4.0.
But I see certain things changed in 4.1.
This is a sequence of REST API calls to install TEXTANALYTICS, that worked with BI 4.1:
curl -u admin:admin -i -H ‘X-Requested-By: ambari’ -d ‘{“ServiceInfo”:{“service_name”:”TEXTANALYTICS”}}’ -X POST http://c2n1.acg.ips:8080/api/v1/clusters/TESTCLUSTER/services
curl -u admin:admin -i -H ‘X-Requested-By: ambari’ -X POST http://c2n1.acg.ips:8080/api/v1/clusters/TESTCLUSTER/services/TEXTANALYTICS/components/TEXTANALYTICS_CLIENT
curl -u admin:admin -i -H ‘X-Requested-By: ambari’ -X POST http://c2n1.acg.ips:8080/api/v1/clusters/TESTCLUSTER/services/TEXTANALYTICS/components/TEXTANALYTICS_MASTER
curl -u admin:admin -i -H ‘X-Requested-By: ambari’ -X POST -d @biginsights_files/service_configs/ta-database-config.json.TESTCLUSTER http://c2n1.acg.ips:8080/api/v1/clusters/TESTCLUSTER/configurations
curl -u admin:admin -i -H ‘X-Requested-By: ambari’ -X PUT -d ‘{ “Clusters” : {“desired_configs”: {“type”: “ta-database-config”, “tag” : “version1436227946689″ }}}’ http://c2n1.acg.ips:8080/api/v1/clusters/TESTCLUSTER
curl -u admin:admin -i -H ‘X-Requested-By: ambari’ -X POST -d @biginsights_files/service_configs/ta-server-config.json.TESTCLUSTER http://c2n1.acg.ips:8080/api/v1/clusters/TESTCLUSTER/configurations
curl -u admin:admin -i -H ‘X-Requested-By: ambari’ -X PUT -d ‘{ “Clusters” : {“desired_configs”: {“type”: “ta-server-config”, “tag” : “version1436227946689″ }}}’ http://c2n1.acg.ips:8080/api/v1/clusters/TESTCLUSTER
curl -u admin:admin -i -H ‘X-Requested-By: ambari’ -X POST -d @biginsights_files/service_configs/ta-service-config.json.TESTCLUSTER http://c2n1.acg.ips:8080/api/v1/clusters/TESTCLUSTER/configurations
curl -u admin:admin -i -H ‘X-Requested-By: ambari’ -X PUT -d ‘{ “Clusters” : {“desired_configs”: {“type”: “ta-service-config”, “tag” : “version1436227946689″ }}}’ http://c2n1.acg.ips:8080/api/v1/clusters/TESTCLUSTER
curl -u admin:admin -i -H ‘X-Requested-By: ambari’ -X POST -d @biginsights_files/service_configs/ta-web-tooling-config.json.TESTCLUSTER http://c2n1.acg.ips:8080/api/v1/clusters/TESTCLUSTER/configurations
curl -u admin:admin -i -H ‘X-Requested-By: ambari’ -X PUT -d ‘{ “Clusters” : {“desired_configs”: {“type”: “ta-web-tooling-config”, “tag” : “version1436227946689″ }}}’ http://c2n1.acg.ips:8080/api/v1/clusters/TESTCLUSTER
curl -u admin:admin -i -H ‘X-Requested-By: ambari’ -X POST -d ‘{“host_components” : [{“HostRoles”:{“component_name”:”TEXTANALYTICS_MASTER”}}] }’ ‘http://c2n1.acg.ips:8080/api/v1/clusters/TESTCLUSTER/hosts?Hosts/host_name=c2n1.acg.ips’
curl -u admin:admin -i -H ‘X-Requested-By: ambari’ -X POST -d ‘{“host_components” : [{“HostRoles”:{“component_name”:”TEXTANALYTICS_CLIENT”}}] }’ ‘http://c2n1.acg.ips:8080/api/v1/clusters/TESTCLUSTER/hosts?Hosts/host_name=c2n1.acg.ips’
curl -u admin:admin -i -H ‘X-Requested-By: ambari’ -X POST -d ‘{“host_components” : [{“HostRoles”:{“component_name”:”TEXTANALYTICS_CLIENT”}}] }’ ‘http://c2n1.acg.ips:8080/api/v1/clusters/TESTCLUSTER/hosts?Hosts/host_name=c2n2.acg.ips’
curl -u admin:admin -i -H ‘X-Requested-By: ambari’ -X POST -d ‘{“host_components” : [{“HostRoles”:{“component_name”:”TEXTANALYTICS_CLIENT”}}] }’ ‘http://c2n1.acg.ips:8080/api/v1/clusters/TESTCLUSTER/hosts?Hosts/host_name=c2n3.acg.ips’
curl -u admin:admin -i -H ‘X-Requested-By: ambari’ -X PUT -d ‘{“RequestInfo”: {“context” :”Install TEXTANALYTICS via REST”}, “Body”: {“ServiceInfo”: {“state” : “INSTALLED”}}}’ http://c2n1.acg.ips:8080/api/v1/clusters/TESTCLUSTER/services/TEXTANALYTICS
However, with BI 4.1 the install fails with this error:
Traceback (most recent call last):
File “/var/lib/ambari-agent/cache/stacks/BigInsights/4.0/hooks/before-ANY/scripts/hook.py”, line 34, in <module>
BeforeAnyHook().execute()
File “/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py”, line 218, in execute
method(env)
File “/var/lib/ambari-agent/cache/stacks/BigInsights/4.0/hooks/before-ANY/scripts/hook.py”, line 30, in hook
setup_users()
File “/var/lib/ambari-agent/cache/stacks/BigInsights/4.0/hooks/before-ANY/scripts/shared_initialization.py”, line 70, in setup_users
ignore_failures = params.ignore_groupsusers_create
File “/usr/lib/python2.6/site-packages/resource_management/core/base.py”, line 157, in __init__
self.env.run()
File “/usr/lib/python2.6/site-packages/resource_management/core/environment.py”, line 152, in run
self.run_action(resource, action)
File “/usr/lib/python2.6/site-packages/resource_management/core/environment.py”, line 118, in run_action
provider_action()
File “/usr/lib/python2.6/site-packages/resource_management/core/providers/accounts.py”, line 133, in action_create
shell.checked_call(command, sudo=True)
File “/usr/lib/python2.6/site-packages/resource_management/core/shell.py”, line 70, in inner
result = function(command, **kwargs)
File “/usr/lib/python2.6/site-packages/resource_management/core/shell.py”, line 92, in checked_call
tries=tries, try_sleep=try_sleep)
File “/usr/lib/python2.6/site-packages/resource_management/core/shell.py”, line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File “/usr/lib/python2.6/site-packages/resource_management/core/shell.py”, line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of ‘groupadd ”’ returned 3. groupadd: ” is not a valid group name
Error: Error: Unable to run the custom hook script [‘/usr/bin/python2.6′, ‘/var/lib/ambari-agent/cache/stacks/BigInsights/4.0/hooks/before-ANY/scripts/hook.py’, ‘ANY’, ‘/var/lib/ambari-agent/data/command-152.json’, ‘/var/lib/ambari-agent/cache/stacks/BigInsights/4.0/hooks/before-ANY’, ‘/var/lib/ambari-agent/data/structured-out-152.json’, ‘INFO’, ‘/var/lib/ambari-agent/data/tmp’]
I noticed that the calls to deploy the service return JSON like this (group_name=null, marked in red):
++ curl -u admin:admin -i -H ‘X-Requested-By: ambari’ -X PUT -d ‘{ “Clusters” : {“desired_configs”: {“type”: “ta-database-config”, “tag” : “version1436227946689″ }}}’ http://c2n1.acg.ips:8080/api/v1/clusters/TESTCLUSTER
HTTP/1.1 200 OK
User: admin
Set-Cookie: AMBARISESSIONID=ci78q3l9eogw1t21some3nag5;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/plain
Vary: Accept-Encoding, User-Agent
Content-Length: 840
Server: Jetty(8.1.17.v20150415)
{
“resources” : [
{
“href” : “http://c2n1.acg.ips:8080/api/v1/clusters/TESTCLUSTER/configurations/service_config_versions?service_name=TEXTANALYTICS&service_config_version=1″,
“configurations” : [
{
“clusterName” : “TESTCLUSTER”,
“stackId” : {
“stackName” : “BigInsights”,
“stackVersion” : “4.1”,
“stackId” : “BigInsights-4.1″
},
“type” : “ta-database-config”,
“versionTag” : “version1436227946689″,
“version” : 2,
“serviceConfigVersions” : null,
“configs” : { },
“configAttributes” : { }
}
],
“group_id” : -1,
“group_name” : null,
“service_config_version” : 1,
“service_config_version_note” : null,
“service_name” : “TEXTANALYTICS”
}
]
}+ echo ‘ta-database-config deploy complete.’
Is the “group_name” : null in the segment above somehow related to the error “Execution of ‘groupadd ”’ returned 3. groupadd: ” is not a valid group name”?
I also noticed that when I install BI value adds via the Ambari UI, the services are defined with “group_name=default”, as in this example:
{
“href” : “http://c2n1:8080/api/v1/clusters/TESTCLUSTER/configurations/service_config_versions?service_name=TEXTANALYTICS&service_config_version=1″,
“cluster_name” : “TESTCLUSTER”,
“configurations” : [
{
“Config” : {
“cluster_name” : “TESTCLUSTER”,
“stack_id” : “BigInsights-4.1″
},
“type” : “ta-web-tooling-config”,
“tag” : “version1441258412457″,
“version” : 1,
“properties” : {
“ta.web.tooling.tokenizer.type” : “standard”
},
“properties_attributes” : { }
},
{
“Config” : {
“cluster_name” : “TESTCLUSTER”,
“stack_id” : “BigInsights-4.1″
},
“type” : “ta-server-config”,
“tag” : “version1441258412457″,
“version” : 1,
“properties” : {
“jetty.server.start.port” : “32000”,
“jetty.server.stop.port” : “32005”
},
“properties_attributes” : { }
},
{
“Config” : {
“cluster_name” : “TESTCLUSTER”,
“stack_id” : “BigInsights-4.1″
},
“type” : “ta-service-config”,
“tag” : “version1441258412457″,
“version” : 1,
“properties” : {
“textanalytics_group” : “hadoop”,
“textanalytics_user” : “tauser”
},
“properties_attributes” : {
“final” : {
“textanalytics_user” : “true”,
“textanalytics_group” : “true”
}
}
},
{
“Config” : {
“cluster_name” : “TESTCLUSTER”,
“stack_id” : “BigInsights-4.1″
},
“type” : “ta-database-config”,
“tag” : “version1441258412457″,
“version” : 1,
“properties” : {
“database.create.new” : “Yes”,
“database.hostname” : ” “,
“database.password” : “******”,
“database.port” : “32050”,
“database.username” : “tauser”,
“database.working.dir” : “/usr/ibmpacks/current/text-analytics-web-tooling/catalog-store”
},
“properties_attributes” : {
“database.password” : {
“toMask” : “true”
}
}
}
],
“createtime” : 1441258411832,
“group_id” : -1,
“group_name” : “default”,
“hosts” : [ ],
“is_cluster_compatible” : true,
“is_current” : true,
“service_config_version” : 1,
“service_config_version_note” : “Initial configurations for BigInsights – Text Analytics”,
“service_name” : “TEXTANALYTICS”,
“stack_id” : “BigInsights-4.1″,
“user” : “admin”
}
Would anyone have any idea as to what could be missing on the Ambari side, that it tries to create a group with an empty name?
Thanks,
Julius