Batch Provisioning Guide

Introduction to batch provisioning

batch prov

The batch provisioning tool is provided for you to provision subscribers and subscriber plans into the Tango© Policy system. The subscriber and plan details that are being provisioned are loaded from a CSV file. This CSV file is automatically processed when copied into a configurable file-input directory. Once processed, the CSV input file produces a report and generates a corresponding log file.

The tool must be run as an application running permanently under the control of the process manager. See set up batch provisioning for more details.

Set up batch provisioning

Typically, batch provisioning is set up and installed by Tango support staff. However, there are cases where you might need to set it up yourself. In order to successfully achieve this task, follow the instructions provided here.

If in doubt about whether you should set up batch provisioning or whether you have permissions to set up batch provisioning, contact Tango Support.

Prerequisites and requirements

You must have full permissions (rwxrwx---) for the SPCM.
You must have the root user password for the database(s).
You must have access to all nodes you wish to run batch provisioning.

Installation procedure

The installation procedure outlined here is for an individual node. You must repeat this process for each node on your cluster.

Create the provisioning processing directory (if it does not exist already).
```
mkdir /tango/data/spcm-provision
```
Permissions should be set as drwxrwx--- tango tango spcm-provision

Copy the subscriber provisioning JAR and shell scripts from the SPCM release package into the bin directory.
```
cp spcmBatchProv_pm.sh spcm-subscriber-provision-batch.jar /tango/bin
```
Permissions should be set as -rwxr-x--- tango tango spcmBatchProv_pm.sh and -rw-rw---- tango tango subscriber-provision-batch.jar.

Copy the sample properties file for provisioning from the SPCM release package into the config directory. Customise as needed for your site.
```
cp spcm-subscriber-provision-batch.properties /tango/config/policy/spcm
```
Permissions should be set as -rw-r-x----- tango tango spcm-subscriber-provision-batch.properties. See a sample properties file.

Copy the two SQL scripts for installing SQL stored procedures and resetting batch tables into the scripts directory.
```
cp createAndInitialisePlan.sql /tango/scripts

cp spring-batch-schema-mysql.sql /tango/scripts
```
Permissions should be set as -rw-rw---- tango tango createAndInitialisePlan.sql and -rw-rw---- tango tango spring-batch-schema-mysql.sql.

Run the SQL script to install the stored procedures required for subscriber batch provisioning.
```
mysql -u root -p root < /tango/scripts/createAndInitialisePlan.sql
```

Update the process manager (PM.cfg) to run the application as a managed process. See the sample with corresponding table below for a better understanding of how to add the application as a managed process. See the complete PM.cfg example to view all fields.

[Process 27] (1)

    Description = SPCM Batch Provision (2)

    Process Path = /tango/bin/spcmBatchProv_pm.sh (3)

    Program Name = spcmBatchProv_pm.sh (4)

    Class Name = spcmBatchProv/0/pmClient/0 (5)

    Port = 16050 (6)

    Argument = ‐x10100 ‐jprovision ‐c/tango/config/policy/spcm/spcmsubscriber‐provision‐batch.properties ‐l/tango/logs/policy/spcm/spcm‐subscriberprovision‐batch.log (7)

   ....

Item Field Description

[Process 27]

Add as a process. 27 is an example. Insert the appropriate N+ your PM.cfg requires. This is typically the next available process.

Description

Description of the process.

Process Path

The path where the process lives.

Program Name

The program name. This is typically the name of the .sh file.

Class Name

The class name.

Port

The process port.

Argument

Where you pass any arguments such as configuration (c) or where log files are to be written (l).

In this case, the log files are written to /tango/logs/policy/spcm/spcm‐subscriberprovision‐batch.log.

Run batch provisioning

After you have set up the directories, stored procedures, and scripts needed to execute batch provisioning, you need to provide a CSV file that adheres to a specific format for both subscribers and plans. See CSV file format to ensure your CSV file matches the correct format. Otherwise, you will receive an error in your report file.

This section contains the following sub-sections:

CSV file format
Run the procedure
Report file output
Log file output

CSV file format

The CSV file(s) that are used for batch provisioning must adhere to a certain format. There is a different format for both:

Subscribers
Plans

When ordering the batch files, you should move (or copy) batch files in the order subscriber, plan, subscriber, plan and so on and so forth.

Subscriber CSV format

The subscriber CSV format should contain a subscriber per line with the following fields.

Subscriber,<msisdn>,<imsi>,<subscriberType>,<subscriberClass>,<subscriberStatus>,<language>,<dpsEnabled>,<dpsNotification>,<imei>,<locationZone>,<renewalDayOfMonth>,<eosNotification>,<paygNotification>,<alternateNotificationMsisdn>,<qoscategoryName>,<ruleViolationNotification>

Plan CSV format

The plan CSV format should contain a plan per line with the following fields.

Plan,<msisdn>,<imsi>,<plan_name>,<plan_status>,<current_usage>,<expiry_date>,<plan_info>,<occurrence_count>,<max_occurrence_count>,<remaining_volume>

Run the procedure

Once you have set up batch provisioning, you need to copy or move your CSV file into the /file-input directory. Follow the instructions provided here to run batch provisioning.

Subscribers and plans should only be provisioned on the node with the active database. Do not copy the CSV file onto the node with the backup database.

Prerequisites and requirements

You must have completed the set up batch provisioning.
You must have a CSV file that adheres to the correct format.
You must have full permissions for the SPCM.

Procedure

Move the CSV file(s) into the file-input directory.

mv subscriber-file.csv /tango/data/spcm-provision/file-input/

mv plan-file.csv /tango/data/spcm-provision/file-input/

Check your report file to see if the CSV file was processed successfully or whether you need to troubleshoot.
Check your log file to see if the CSV file was processed successfully or whether you need to troubleshoot.

Report file output

After the batch provisioning job is run, your reports can be found in the /tango/data/spcm-provision/report directory. A sample report with table descriptions are included here to help you get an idea of what to expect when reading a report file.

You can configure a different directory for your reports in the spcm-subscriber-provision-batch.properties file with the spcm.subscriber.provision.batch.report.directory property.

Report file sample

    /tango/data/spcm-provision/report/2MSubs-From-0640000000_1.report

    File Name : 2MSubs-From-0640000000_1.csv (1)
    Start Time : 2013-10-02T14:06:05 (2)
    Read Count : 2000000 (3)
    Write Count : 2000000 (4)
    Skip Count : 0 (5)
    Step Exit Status : COMPLETED (6)
    Batch Status : COMPLETED (7)

Report file field descriptions

Number Field Description Sample

File Name

The name of the file processed.

2MSubs-From-0640000000_1.csv

Start Time

The date and time the file was processed.

This is in ISO 8601 format.

2013-10-02T14:06:05

Read Count

The number of records (rows) read from the file.

2000000

Write Count

The number of records processed and executed.

2000000

Skip Count

The number of records skipped due to error.

0

Step Exit Status

The exit status for the tool itself.

possible values

COMPLETED = completed successfully
COMPLETED WITH SKIPS = completed with some records (rows) missed
FAILED = batch provisioning failed

COMPLETED

Batch Status

The exit status for the batch processing.

possible values

COMPLETED = completed successfully
FAILED = batch provisioning failed

COMPLETED

Log file output

After the batch provisioning job is run, your logs can be found at /tango/logs/policy/spcm/spcm-subscriber-provision-batch.log directory. A sample log file is included here to help you get an idea of what to expect when reading the log file.

You can configure a different directory for your logs in the PM.cfg file by passing the -l argument in the Argument section.

The file is rolled over at midnight with a date/time stamp.

Log file sample

$ less /tango/logs/policy/spcm/spcm-subscriber-provision-batch.log

2019-12-12 12:40:10,028 INFO  [task-scheduler-10] o.s.integration.file.FileReadingMessageSource : Created message: [GenericMessage [payload=/tango/data/spcm-provision/file-input/CUSTOMER_20191212_0000.csv, headers={id=8946bc3f-26f9-ae13-a2cd-e932f746170e, timestamp=1504262410027}]]

2019-12-12 12:40:10,099 INFO  [task-scheduler-10] o.s.i.file.FileWritingMessageHandler : Failed to move file '/tango/data/spcm-provision/file-input/CUSTOMER_20191212_0000.csv'. Using copy and delete fallback.

2019-12-12 12:40:10,269 INFO  [task-scheduler-10] o.s.b.core.launch.support.SimpleJobLauncher : Job: [FlowJob: [name=batchProvision]] launched with the following parameters: [{file.input=/tango/data/spcm-provision/file-processing/20190901124010059_CUSTOMER_20191212_0000.csv, timestamp=1504262410114}]

2019-12-12 12:40:10,289 INFO  [task-scheduler-10] c.t.s.s.p.b.l.BatchProvisionJobListener : Job start time : [Fri Sep 01 12:40:10 CEST 2019] File name : [/tango/data/spcm-provision/file-processing/20190901124010059_CUSTOMER_20191212_0000.csv]

2019-12-12 12:40:10,306 INFO  [task-scheduler-10] o.s.batch.core.job.SimpleStepHandler : Executing step: [provisionStep]

2019-12-12 12:40:17,222 INFO  [provisionPlan-11] c.t.s.s.p.batch.listeners.DebugChunkListener : Number of chunks completed: count=10, job=batchProvision,step=provisionStep

2019-12-12 12:40:17,925 INFO  [provisionPlan-8] c.t.s.s.p.batch.listeners.DebugChunkListener : Number of chunks completed: count=20, job=batchProvision,step=provisionStep
<snip>

2019-12-12 12:40:22,873 INFO  [provisionPlan-11] c.t.s.s.p.batch.listeners.DebugChunkListener : Number of chunks completed: count=90, job=batchProvision, step=provisionStep

2019-12-12 12:40:22,885 INFO  [provisionPlan-3] c.t.s.s.p.batch.listeners.DebugChunkListener : Number of chunks completed: count=100, job=batchProvision,step=provisionStep

2019-12-12 12:40:22,886 INFO  [task-scheduler-10] c.t.s.s.p.batch.listeners.ReportStepListener : summary StepExecution: id=1003, version=102, name=provisionStep, status=COMPLETED, exitStatus=COMPLETED, readCount=5000, filterCount=0, writeCount=5000 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=101, rollbackCount=0

2019-12-12 12:40:22,895 INFO  [task-scheduler-10] c.t.s.s.p.b.l.BatchProvisionJobListener : Job start time : [Fri Sep 01 12:40:10 CEST 2019], Job end time : [Fri Sep 01 12:40:22 CEST 2019], Duration in seconds : [12], FileName : [/tango/data/spcm-provision/file-processing/20190901124010059_CUSTOMER_20191212_0000.csv]

2019-12-12 12:40:22,896 INFO  [task-scheduler-10] o.s.b.core.launch.support.SimpleJobLauncher : Job: [FlowJob: [name=batchProvision]] completed with the following parameters: [{file.input=/tango/data/spcm-provision/file-processing/20190901124010059_CUSTOMER_20191212_0000.csv, timestamp=1504262410114}] and the following status: [COMPLETED]

Batch provisioning error help

Errors are uncommon when running batch provisioning if the CSV file is formatted correctly, and the tool is set up and configured to run as an application under the control of the process manager. However, if you do encounter an issue with a particular error, it is best to know how to check those errors.

This section does not, of course, contain all of the possible errors that should be checked and is simply a starting point for understanding any errors you might receive while batch provisioning subscribers and plans.

Where do I find batch processing errors?

Errors are sent to the the <csv-file-name>.error file. The .error file uses the same name as the CSV file you provided. An example is shown below with the false (1) errors being shown.

Navigate to this file through your machine’s directory structure or if using linux, you can use the less command. Typically the file is found in the /tango/data/spcm-provision/error/ folder. However, this directory is configureable via the spcm.subscriber.provision.batch.error.directory property in the properties file which is found in the /config folder.

$ less /tango/data/spcm-provision/error/20191212112705015_CUSTOMER_20191212_1000.error

Subscriber,096467134885,176093556911209,UnKnown,OperatorASubscriber,Active,Spanish,false,false,,,1,false,false,, (1)
Plan,096467134885,176093556911209,Plan-Name-A,Active,,2017-03-31T22:00:00,,null,1,
Plan,096467134885,176093556911209,Plan-Name-Other,Active,,2017-03-31T22:00:00,,null,1,

Check the CSV file is formatted correctly

One of the more common errors is that the subscriber (or plan) was not properly defined in the CSV file. If the CSV file is mostly formatted correctly, you might only have one or two subscribers that were not entered correctly.

Check that your CSV file matches the correct CSV file format.

options

Navigate to the file-processed directory of the .csv file that you provided for batch provisioning and check the subscriber.

Run an egrep command with the subscriber MSISDN pointed at the directory where your .csv file resides. An example of this command is shown below.

$ egrep 096467134885 /tango/data/spcm-provision/file-processed/20191212112705015_CUSTOMER_20191212_1000.csv

Subscriber,096467134885,176093556911209,Unknown,OperatorASubscriber,active,Spanish,0,0,,,1,0,0,
Plan,096467134885,176093556911209,Plan-Name-A,active,,,
Plan,096467134885,176093556911209,Plan-Name-Other,active,,,

Typically the file-processed directory is set at /tango/data/spcm-provision/file-processed. However, this is configurable in the properties file so check your file to make sure you are looking in the right place.

Check if the subscriber is already in the SPCM

If you receive an error for a particular subscriber in batch provisioning, it is possible that the subscriber already exists in the SPCM. In this case, use the SPCM API to check and see if the subscriber already exists. If the subscriber does exist, go on to see if a plan was purchased before you attempted the batch provisioning. If a plan was purchased (see purchaseTimestamp parameter in the JSON response) before batch provisioning occurred. This is why the subscriber threw an error.

See the Get subscriber plans and Get subscriber info API operations for instructions on how to do this.