Integration of Templates using JDF

Jamcracker enables developers and business an easy way to integrate AWS CloudFormation templates, Azure ARM templates and various API’s (like Jenkins , REST API) into Jamcracker platform using Jamcracker DevOps Framework (JDF) scripts. By doing this, you can create a service along with an offer in JSDN or on-board the service into a store Catalog. However, before you start, you must know how to add resource scripts through Database and define variables. To add resource scripts through the Database and define variables, you must have:

  1. Knowledge of Unix Commands AND SQL Queries
  2. Postgres Database Details
    • Database Server IP
    • Server Root User Credentials
    • Database Username and Password
  3. Application Server Details
    • App Server IP
    • Server Root User Credentials
  4. Permission to Access the Application Server and Database Server

Preparing Seed Data:

  • List of available resource types, the resource type that you insert should be different from the one already available.
  • Alternatively the following select query can also be executed to check the uniqueness of the resource type

    select count(*) from JCO_RESOURCE_TYPE where type=’Resource type’;

    If the result is 0 – The resource type entered is unique.

    If the result is 1 – The resource type already exists.

To insert a new resource type into the DB

<!-- Change sets starts -->
<!-- To insert a new resource type to the DB starts -->

<changeSet author="7.12.6.2" id="ResourceType_route53_adding_201809041200"> <!—Replace with unique Resource type and timestamp--> 
 	<insert tableName="JCO_RESOURCE_TYPE">
                 <column name="ID" valueComputed="(SELECT MAX(ID)+1 FROM JCO_RESOURCE_TYPE)"/>
                 <column name="TYPE" value="route53"/> <!—Replace with Resource type-- >
                 <column name="DESCRIPTION" value="route53"/> <!—Replace with Resource type-- >
                 <column name="CLASSNAME" value="com.jamcracker.commons.entities.iaas.Generic"/>
                 
   </insert>
</changeSet>

<changeSet author="7.12.6.2" id="ResourceType_route53_msg_201809041200">  <!—Replace with unique Resource type and timestamp-->
    <insert tableName="JCP_UI_METADATA">
		    <column name="RESOURCE_KEY" value="jsdn.store.resourcescript.resourcetype.route53"/> <!—Replace with Resource type, Mandatory should be same as the one mentioned in the above changeset-- >
		    <column name="ORGANIZATION_ID" valueNumeric="1000"/>
		    <column name="LANGUAGE_CODE" value="en_US"/>
		    <column name="RESOURCE_TYPE" value="M"/>
		    <column name="RESOURCE_VALUE" value="Route53"/> <!—Replace with Resource type, Value that appears in the UI-- >
	</insert>
</changeSet>
<!-- Change sets ends -->
<!-- Insert a new resource type to the DB Ends -->

To update JCI_PROVIDER_RESOURCE_TEMPLATE_S sequence

<!-- To update JCI_PROVIDER_RESOURCE_TEMPLATE_S sequence starts -->

<changeSet id="RESOURCE_TEMPLATE_SEQ_UPDATE_201809071110" author="7.12.2" dbms="postgresql"> <!—Replace with the time stamp --> 
    <comment>JCI_PROVIDER_RESOURCE_TEMPLATE.JCI_PROVIDER_RESOURCE_TEMPLATE_S update</comment>
    <sql>select setval('JCI_PROVIDER_RESOURCE_TEMPLATE_S', cast((select max(PROVIDER_RESOURCE_TEMPLATE_ID)+1 from JCI_PROVIDER_RESOURCE_TEMPLATE) as bigint))</sql>
</changeSet>
<!-- To update JCI_PROVIDER_RESOURCE_TEMPLATE_S sequence ends -->

To add new resource template into the DB

<!-- Change sets starts-->
<!-- To add new resource template into the DB Starts -->
1. List of available resource templates, the resource template code that you insert should be different from the one already available. 
2. Alternatively the following select query can also be executed to check the uniqueness of the template code. 
select count(*) from  JCI_PROVIDER_RESOURCE_TEMPLATE where TEMPLATE_NAME=’Template Code’;	
If the result is 0 – The resource type entered is unique.
	If the result is 1 – The resource type already exists.

<changeSet author="7.12.6.2" id="Template_createZone_adding_201809041200"> <!—Replace with unique Resource template name and timestamp-->
 	<insert tableName="JCI_PROVIDER_RESOURCE_TEMPLATE">
                 <column name="PROVIDER_RESOURCE_TEMPLATE_ID" valueComputed="${nextvalue_JCI_PROVIDER_RESOURCE_TEMPLATE_S}"/>
                 <column name="RESOURCE_TEMPLATE_TYPE_ID" valueComputed="(SELECT resource_template_type_id FROM jci_resource_template_type  WHERE resource_template_type_code='ansible')"/>
                 <column name="PROVIDER_CODE" value="aws"/> <!—Replace with the provider code, an ondemand provider needs to exist on JSDN, if the provider does not exist please onboard a ondemand provider-->
                 <column name="ACTOR_ID" value="1000"/>
                 <column name="RESOURCE_TYPE_ID" valueComputed="(SELECT ID FROM JCO_RESOURCE_TYPE WHERE TYPE='route53')"/> <!— Replace with the Resource type -- > 
                 <column name="TEMPLATE_ACTION_CODE" value="create"/>
                 <column name="TEMPLATE_NAME" value="createZone"/> <! – Replace with Template Name -->
                 <column name="TEMPLATE_CONTENT" value="Create Zone"/> <! –Replace with Template Name -->
                 <column name="STATUS" value="A"/>
                 <column name="CREATED_DATE" valueDate="${SYSTEMDATE}"/>
                 <column name="CREATED_BY" value="1002"/>
                 <column name="UPDATED_DATE" valueDate="${SYSTEMDATE}"/>
                 <column name="UPDATED_BY" value="1002"/>                
   </insert>
</changeSet>
<!-- Adding new resource template into the DB Ends -->

To update resource script to the existing template into DB starts

<changeSet id="Template_createZone_update_script_201809041200" author="7.12.6.2"> <!—
Replace  with unique Resource template name and timestamp provided above in the previous query-->
	<comment>updating JCI_PROVIDER_RESOURCE_TEMPLATE  records . . .</comment>
    	<sql splitStatements="false">
update JCI_PROVIDER_RESOURCE_TEMPLATE set TEMPLATE_CONTENT='- hosts: localhost
  connection: local
  vars:
   zone: ""
  tasks:
   - name: Creating {{zone}} zone
     route53_zone:
      aws_access_key: "{{ accessKey }}"
      aws_secret_key: "{{ secretKey }}"
      zone: "{{ zone }}"
      state: 'present'
      comment: This zone created for {{zone}}'  <!—Replace with the Custom template script --> 
where PROVIDER_CODE='aws' and ACTOR_ID =1000 and TEMPLATE_ACTION_CODE='create' and TEMPLATE_NAME='createZone' <! – Replace with the template name, mentioned above-->
    </sql>
</changeSet>

<!—Update resource script to existing template into DB ends -->

To update resource script variables in json format to existing template into DB

<!-- To update resource script variables in json format to existing template into DB starts -->


<changeSet id="Template_createZone_update_variables_201809041200" author="7.12.6.2"> <!—Replace with the template name and timestamp-->
        <comment>updating JCI_PROVIDER_RESOURCE_TEMPLATE for Azure stack tenant mapping Stub</comment> 
<sql splitStatements="false">
update JCI_PROVIDER_RESOURCE_TEMPLATE set SCRIPT_INPUT_JSON='{"optional":[],"layout":"double","required":[{"help":"jsdn.resourcescript.help.zone","validation":"jsdn.resourcescript.validation.zone","value":"","label":"jsdn.resourcescript.lable.zone","key":"zone","validationmsg":"jsdn.resourcescript.validationmsg.zone"}]}' 
<!—Construct a JSON with Optional or Required fields-->
where PROVIDER_CODE='aws' and ACTOR_ID =1000 and TEMPLATE_ACTION_CODE='create' and TEMPLATE_NAME='createZone' <!—Replace with the Provider Code and Template Name defined above-->
    </sql>
</changeSet>

<!-- Updating resource script variables in json format to existing template into DB ends -->

To insert resource script message for existing template into DB

<!—To insert resource script message for existing template into DB starts -->
<changeSet author="7.12.6.2" id="Template_createZone_msges_201809041200"> <!—Replace with the template name and timestamp-->

    <insert tableName="JCP_UI_METADATA">
		    <column name="RESOURCE_KEY" value="jsdn.store.resourcescript.template.createZone"/> <!—Replace with the template name defined above-->
		    <column name="ORGANIZATION_ID" valueNumeric="1000"/>
		    <column name="LANGUAGE_CODE" value="en_US"/>
		    <column name="RESOURCE_TYPE" value="M"/>
		    <column name="RESOURCE_VALUE" value="Create Zone"/> <!—Replace with template name--> 
	</insert>

    <insert tableName="JCP_UI_METADATA">
		    <column name="RESOURCE_KEY" value="jsdn.resourcescript.help.zone"/> <!—Replace with the variable that is part of the script--> 
		    <column name="ORGANIZATION_ID" valueNumeric="1000"/>
		    <column name="LANGUAGE_CODE" value="en_US"/>
		    <column name="RESOURCE_TYPE" value="M"/>
		    <column name="RESOURCE_VALUE" value="Help message for zone"/> <!—Replace with the  resource value for the variable that is part of the script-->
	</insert>
    <insert tableName="JCP_UI_METADATA">
		    <column name="RESOURCE_KEY" value="jsdn.resourcescript.validation.zone"/> <!—Replace with the variable that is part of the script--> 

		    <column name="ORGANIZATION_ID" valueNumeric="1000"/>
		    <column name="LANGUAGE_CODE" value="en_US"/>
		    <column name="RESOURCE_TYPE" value="M"/>
		    <column name="RESOURCE_VALUE" value="Validation regular expression for zone"/> <!—Replace with the  resource value for the variable that is part of the script-->

	</insert>
    <insert tableName="JCP_UI_METADATA">
		    <column name="RESOURCE_KEY" value="jsdn.resourcescript.lable.zone"/> <!—Replace with the variable that is part of the script--> 

		    <column name="ORGANIZATION_ID" valueNumeric="1000"/>
		    <column name="LANGUAGE_CODE" value="en_US"/>
		    <column name="RESOURCE_TYPE" value="M"/>
		    <column name="RESOURCE_VALUE" value="Zone"/> <!—Replace with the  resource value for the variable that is part of the script-->

	</insert>
    <insert tableName="JCP_UI_METADATA">
		    <column name="RESOURCE_KEY" value="jsdn.resourcescript.validationmsg.zone"/> <!—Replace with the variable that is part of the script--> 

		    <column name="ORGANIZATION_ID" valueNumeric="1000"/>
		    <column name="LANGUAGE_CODE" value="en_US"/>
		    <column name="RESOURCE_TYPE" value="M"/>
		    <column name="RESOURCE_VALUE" value="Validation error message for zone"/> <!—Replace with the  resource value for the variable that is part of the script-->

	</insert>
</changeSet>

<!-- Inserting resource script message for existing template into DB ends -->
<!-- Change sets ends -->

Once you have the above seed data ready

3. Update the below property in /d01/jboss-as-7.1.1.Final/install_home/build.properties files build.components.db=resourcescript

4. Check proper DB details updated in /d01/jboss-as-7.1.1.Final/install_home/installation.properties or not – Optional.

5. Goto /d01/jboss-as-7.1.1.Final/install_home/ folder and execute below target

ant updateComponentDB

6. Restart jboss server(s) for all the UI messages to reflect.

AWS - CloudFormation

How does it work with AWS (CloudFormation) template?

When you create a stack, AWS CloudFormation makes underlying service calls to AWS to provision and configure your resources.

Note that AWS CloudFormation can perform only actions that you have permission to do. For example, to create EC2 instances by using AWS CloudFormation, you need permissions to create instances. You'll need similar permissions to terminate instances when you delete stacks with instances. Refer to AWS Identity and Access Management (IAM) to manage permissions.

The calls that AWS CloudFormation makes are all declared by your template. For example, suppose you have a template that describes an EC2 instance with a t1.micro instance type. When you use that template to create a stack, AWS CloudFormation calls the Amazon EC2 create instance API and specifies the instance type as t1.micro.

You can write an AWS CloudFormation template (a JSON or YAML-formatted document) in a text editor. You can also choose to use a provided template. The template describes the resources you want, region and their settings. For example, suppose you want to create an EC2 instance. Your template can declare an EC2 instance and describe its properties, as shown in the following example:

Sample JSON Template Syntax

 {
 "AWSTemplateFormatVersion" : "2010-09-09",
 "Description" : "A simple EC2 instance",
 "Resources" : {
 "MyEC2Instance" : {
 "Type" : "AWS::EC2::Instance",
 "Properties" : {
 "ImageId" : "ami-0ff8a91507f77f867",
 "InstanceType" : "t1.micro"
 }
 }
 }
 }

Save the template locally or in an S3 bucket. If you created a template, save it with any file extension like .template.

Create an AWS CloudFormation stack by specifying the location of your template file, such as a path on your local computer or an Amazon S3 URL. If the template contains parameters, you can specify input values when you create the stack. Parameters enable you to pass in values to your template so that you can customize your resources each time you create a stack.

Note: If you specify a template file stored locally, AWS CloudFormation uploads it to an S3 bucket in your AWS account. AWS CloudFormation creates a bucket for each region in which you upload a template file. The buckets are accessible to anyone with Amazon Simple Storage Service (Amazon S3) permissions in your AWS account. If a bucket created by AWS CloudFormation is already present, the template is added to that bucket.

You can use your own bucket and manage its permissions by manually uploading templates to Amazon S3. Then whenever you create or update a stack, specify the Amazon S3 URL of a template file.

AWS CloudFormation provisions and configures resources by making calls to the AWS services that are described in your template.

After all the resources have been created, AWS CloudFormation reports that your stack has been created. You can then start using the resources in your stack. If stack creation fails, AWS CloudFormation rolls back your changes by deleting the resources that it created.

Now, create a yaml file based on your requirement and copy the content in JDF Template and follow the steps.

Sample Yaml File (WordPress Instance):

- hosts: localhost
 gather_facts: no 
 vars:
 aws_region: "us-east-1"
 tasks:
 - name: create a cloudformation stack
 cloudformation:
 stack_name: "ansible-cloudformation"
 state: "present"
 region: "'us-east-1','us-east-1','us-east-1','us-east-1'"
 region: {\"AllowedValues\": [ \"t1.micro\", \"t2.nano\", \"t2.micro\", \"t2.small\", \"t2.medium\", \"t2.large\"]}
 region: {"AllowedValues": [ "t1.micro", "t2.nano", "t2.micro", "t2.small", "t2.medium", "t2.large"]}
 disable_rollback: true
 template_url: 
 #template: "/home/Downloads/AWSCloudFormation-samples/WordPress_Single_Instance.template"
 template_parameters:
 KeyName: "Payeeaccountpem"
 #DiskType: "ephemeral"
 DBName: "jccfdb"
 DBUser: "jccfdbuser"
 DBPassword: "jccfdbuser"
 DBRootPassword: "jccfdbuser"
 InstanceType: "t2.micro"
 #ClusterSize: 3
 tags:
 Stack: "jc-ansible-cloudformation"

Azure - ARMs

How does it work with Azure (ARMs) template?

Creating your first Azure Resource Manager template using Azure portal, and the process of editing and deploying the template from the Azure portal. Resource Manager templates are JSON files that define the resources you need to deploy for your solution. However, before you begin, you must have a valid Azure account.

Generate a template using the portal

In this section, you create a storage account using the Azure portal. Before you deploy the storage account, you have the option to explore the template generated by the portal based on your configurations. You can save the template and reuse it in the future.

  1. Sign in to the Azure portal.
  2. Select Create a resource > Storage > Storage account - blob, file, table, queue.

  3. Enter the following information.
    • Resource group: create a new Azure resource group with the name of your choice. On the screen shot, the resource group name is mystorage1016rg.
    • Name: give your storage account a unique name. On the screen shot, the name is mystorage1016.

    You can use the default values for the rest of the properties.

    Note: Some of the exported templates require some edits before you can deploy them.
  4. Select Review + create on the bottom of the screen.
  5. Select Download a template for automation on the bottom of the screen. The portal shows the generated template:

    The main pane shows the template. It is a JSON file with four top-level elements - schema, contentVersion, parameters and resources. For more information, see Understand the structure and syntax of Azure Resource Manager Templates

    There are six parameters defined. One of them is called storageAccountName. The second highlighted part shows how to use this parameter in the template. In the next section, you edit the template to use a generated name for the storage account.

    In the template, one Azure resource is defined. The type is [Microsoft.Storage/storageAccounts]. See how the resource is defined, and the definition structure.

  6. Select Download. Save template.json from the downloaded package to your computer. In the next section, you use a template deployment tool to edit the template.
  7. Select the Parameter tab to see the values you provided for the parameters. Write down these values, you need them in the next section when you deploy the template.

Using both the template and the parameters files, you can create an Azure storage account.

Edit and deploy the template

The Azure portal can be used to perform some basic template editing. In this quickstart, you use a portal tool called Template Deployment. To edit a more complex template, consider using Visual Studio Code which provides richer edit functionalities.

Azure requires that each Azure service has a unique name. The deployment fails if you enter a storage account name that already exists. To avoid this issue, you can use a template function call uniquestring() to generate a unique storage account name.

  1. In the Azure portal, select Create a resource.
  2. In Search the Marketplace, type template deployment, and then press ENTER.
  3. Select Template deployment.

  4. Select Create.
  5. Select Build your own template in the editor.
  6. Select Load file, and then follow the instructions to load template.json you downloaded in the last section.
  7. Add one variable as shown in the following screenshot:

    JSON

    "storageAccountName": "[concat(uniquestring(resourceGroup().id), 'standardsa')]"

    Two functions are used here: concat() and uniqueString().

  8. Remove the storageAccountName parameter highlighted in the previous screenshot.
  9. Update the name element of the Microsoft.Storage/storageAccounts resource to use the newly defined variable instead of the parameter:

    JSON

    "name": "[variables('storageAccountName')]",

    The final template shall look like:

    JSON
    {
     "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
     "contentVersion": "1.0.0.0",
     "parameters": {
     "location": {
     "type": "string"
     },
     "accountType": {
     "type": "string"
     },
     "kind": {
     "type": "string"
     },
     "accessTier": {
     "type": "string"
     },
     "supportsHttpsTrafficOnly": {
     "type": "bool"
     }
     },
     "variables": {
     "storageAccountName": "[concat(uniquestring(resourceGroup().id), 'standardsa')]"
     },
     "resources": [
     {
     "name": "[variables('storageAccountName')]",
     "type": "Microsoft.Storage/storageAccounts",
     "apiVersion": "2018-07-01",
     "location": "[parameters('location')]",
     "properties": {
     "accessTier": "[parameters('accessTier')]",
     "supportsHttpsTrafficOnly": "[parameters('supportsHttpsTrafficOnly')]"
     },
     "dependsOn": [],
     "sku": {
     "name": "[parameters('accountType')]"
     },
     "kind": "[parameters('kind')]"
     }
     ],
     "outputs": {}
    }
  10. Select Save.
  11. Enter the following values:
    • Resource group: name your resource group with a unique name.
    • Location: select a location for the resource group.
    • Location: select a location for the storage account. You can use the same location as the resource group.
    • Account Type: Enter Standard_LRS for this quickstart.
    • Kind: Enter StorageV2 for this quickstart.
    • Access Tier: Enter Hot for this quickstart.
    • Https Traffic Only Enabled. Select true for this quickstart.
    • I agree to the terms and conditions stated above: (select)

    Here is a screenshot of a sample deployment:

  12. Select Purchase.
  13. Select the bell icon (notifications) from the top of the screen to see the deployment status. Wait until the deployment is completed.

  14. Select Go to resource group from the notification pane. You shall see a screen similar to:

You can see the deployment status was successful, and there is only one storage account in the resource group. The storage account name is a unique string generated by the template.

Now, create a yaml file based on your requirement and copy the content in JDF Template and follow the steps.

Sample Yaml File (WordPress Instance):

- hosts: localhost
 gather_facts: no 
 vars:
 aws_region: "us-east-1"
 tasks:
 - name: create a cloudformation stack
 cloudformation:
 stack_name: "ansible-cloudformation"
 state: "present"
 region: "'us-east-1','us-east-1','us-east-1','us-east-1'"
 region: {\"AllowedValues\": [ \"t1.micro\", \"t2.nano\", \"t2.micro\", \"t2.small\", \"t2.medium\", \"t2.large\"]}
 region: {"AllowedValues": [ "t1.micro", "t2.nano", "t2.micro", "t2.small", "t2.medium", "t2.large"]}
 disable_rollback: true
 template_url: 
 #template: "/home/Downloads/AWSCloudFormation-samples/WordPress_Single_Instance.template"
 template_parameters:
 KeyName: "Payeeaccountpem"
 #DiskType: "ephemeral"
 DBName: "jccfdb"
 DBUser: "jccfdbuser"
 DBPassword: "jccfdbuser"
 DBRootPassword: "jccfdbuser"
 InstanceType: "t2.micro"
 #ClusterSize: 3
 tags:
 Stack: "jc-ansible-cloudformation"

Jenkins API

How does it work with Jenkins APIs?

Here is the sample Jenkins script and make the necessary changes. Copy the content in JDF Template and follow the steps.

Create a yaml file based on your requirement and copy the content in JDF Template and follow the steps.

Sample Jenkins API Script:

- hosts: localhost
  gather_facts: no 
  vars:
   username: "jstpldev" # Username is jstpldev
   authenticationToken: "scm@789"  # Password is scm@789
   jenkinsURL: "172.31.10.222"  # URL should be 172.31.10.222
   triggerBuild: "" # Place as 'view/VADS/job/VADS_impResrApp7.12.4.1'
   success_check_max_retries: "" # Suggested value is 60
   poll_delay_seconds: "" # Suggested value is 60
   
  tasks:
  - name: Call to get Jenkins-Crumb
    uri:
       url: 'http://{{ jenkinsURL }}/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)'
       user: "{{ username }}"
       password: "{{ authenticationToken }}"
       force_basic_auth: yes
       return_content: yes
    register: crumb_response1
    
  - name:               Set crumb_response in variable
    set_fact:
      crumb_response:           "{{ crumb_response1.content.split(':')[1] }}"
    
  - name:               Call Jenkins
    uri:
       url: http://{{ jenkinsURL }}/{{ triggerBuild }}/build?delay=0sec
       method: POST
       user: "{{ username }}"
       password: "{{ authenticationToken }}"
       force_basic_auth: yes
       return_content: yes
       status_code: 200,201
       headers:
         Jenkins-Crumb: "{{ crumb_response }}"
    register: json_response
    
  - name: debug jenkins request result
    debug:
        var: json_response
        verbosity: 1
    
  - name: query jenkins job queue to see if it has been able to start a build
    block:
    - name: poll jenkins JOB QUEUE every 3 seconds for configured number of attempts to get the number from the job queue
      uri:
        force_basic_auth: yes
        headers:
          Content-Type: "application/json"
        method: GET
        password: "{{ authenticationToken }}"
        return_content: yes
        timeout: 3
        url: "{{json_response.location}}api/json"
        user: "{{ username }}"
      register: poll_result
      until: poll_result|json_query('json.executable.number')|default(0)|int > 0
      retries: '{{ success_check_max_retries }}'
      delay: '{{ poll_delay_seconds }}'
              
  - name: now poll jenkins JOB every 3 seconds for configured number of attempts to check if the job was actually successful
    uri:
        force_basic_auth: yes
        headers:
          Content-Type: "application/json"
        method: GET
        password: "{{ authenticationToken }}"
        return_content: yes
        timeout: 3
        url: http://{{ jenkinsURL }}/{{ triggerBuild }}/{{poll_result.json.executable.number}}/api/json
        user: "{{ username }}"
    register: build_job_result
    failed_when: build_job_result|json_query('json.result') == 'FAILED'
    until: build_job_result|json_query('json.result') == 'SUCCESS'
    retries: '{{ success_check_max_retries }}'
    delay: '{{ poll_delay_seconds }}'