Batch Provisioning Guide

Introduction to batch provisioning

batch prov

The batch provisioning tool is provided for you to provision subscribers and subscriber plans into the Tango© Policy system. The subscriber and plan details that are being provisioned are loaded from a CSV file. This CSV file is automatically processed when copied into a configurable file-input directory. Once processed, the CSV input file produces a report and generates a corresponding log file.

The tool must be run as an application running permanently under the control of the process manager. See set up batch provisioning for more details.

Set up batch provisioning

Typically, batch provisioning is set up and installed by Tango support staff. However, there are cases where you might need to set it up yourself. In order to successfully achieve this task, follow the instructions provided here.

If in doubt about whether you should set up batch provisioning or whether you have permissions to set up batch provisioning, contact Tango Support.

Prerequisites and requirements

You must have full permissions (rwxrwx---) for the SPCM.
You must have the root user password for the database(s).
You must have access to all nodes you wish to run batch provisioning.

Installation procedure

The installation procedure outlined here is for an individual node. You must repeat this process for each node on your cluster.

Create the provisioning processing directory (if it does not exist already).
```
mkdir /tango/data/spcm-provision
```
Permissions should be set as drwxrwx--- tango tango spcm-provision

Copy the subscriber provisioning JAR and shell scripts from the SPCM release package into the bin directory.
```
cp spcmBatchProv_pm.sh spcm-subscriber-provision-batch.jar /tango/bin
```
Permissions should be set as -rwxr-x--- tango tango spcmBatchProv_pm.sh and -rw-rw---- tango tango subscriber-provision-batch.jar.

Copy the sample properties file for provisioning from the SPCM release package into the config directory. Customise as needed for your site.

cp spcm-subscriber-provision-batch.properties /tango/config/policy/spcm

Permissions should be set as -rw-r-x----- tango tango spcm-subscriber-provision-batch.properties. See the sample properties file.

hibernate.dialect=org.hibernate.dialect.MySQL5InnoDBDialect
jdbc.driverClassName=com.mysql.jdbc.Driver
jdbc.url=jdbc:mysql://localhost/pmsdb?useUnicode=true&characterEncoding=utf8useUnicode=true&characterEncoding=utf8
jdbc.username=root
jdbc.password=root
jdbc.connection.pool.max.active.connections=2000
jdbc.connection.pool.min.idle.connections=100
jdbc.connection.pool.max.idle.connections=200
jdbc.validation.query=SELECT 1;
spcm.subscriber.default.class.name=Standard
spcm.subscriber.provision.batch.file.input.directory=/tango/data/spcm-provision/file-input
spcm.subscriber.provision.batch.file.processing.directory=/tango/data/spcm-provision/file-processing
spcm.subscriber.provision.batch.file.processed.directory=/tango/data/spcm-provision/file-processed
spcm.subscriber.provision.batch.file.batch.db.import.directory=/tango/data/spcm-provision/db-file-import
spcm.subscriber.provision.batch.error.directory=/tango/data/spcm-provision/error
spcm.subscriber.provision.batch.report.directory=/tango/data/spcm-provision/report
spcm.subscriber.provision.batch.file.filename.pattern=*.csv
spcm.subscriber.provision.batch.file.poller.cron=*/5 * * * * *
spcm.subscriber.provision.batch.plan.commit.interval=1000
spcm.subscriber.provision.batch.thread.executor.core.pool.size=2422
spcm.subscriber.provision.batch.thread.executor.max.pool.size=48
spcm.subscriber.provision.batch.plan.schedule.commit.interval=100
spcm.subscriber.provision.batch.validate.split.skip.limit = 200

Copy the two SQL scripts for installing SQL stored procedures and resetting batch tables into the scripts directory.
```
cp createAndInitialisePlan.sql /tango/scripts

cp spring-batch-schema-mysql.sql /tango/scripts
```
Permissions should be set as -rw-rw---- tango tango createAndInitialisePlan.sql and -rw-rw---- tango tango spring-batch-schema-mysql.sql.

Run the SQL script to install the stored procedures required for subscriber batch provisioning.
```
mysql -u root -p root < /tango/scripts/createAndInitialisePlan.sql
```

Update the process manager (PM.cfg) to run the application as a managed process. See the sample with corresponding table below for a better understanding of how to add the application as a managed process. See the complete PM.cfg example to view all fields.

[Process 27] (1)

    Description = SPCM Batch Provision (2)

    Process Path = /tango/bin/spcmBatchProv_pm.sh (3)

    Program Name = spcmBatchProv_pm.sh (4)

    Class Name = spcmBatchProv/0/pmClient/0 (5)

    Port = 16050 (6)

    Argument = ‐x10100 ‐jprovision ‐c/tango/config/policy/spcm/spcmsubscriber‐provision‐batch.properties ‐l/tango/logs/policy/spcm/spcm‐subscriberprovision‐batch.log (7)

   ....

Item Field Description

[Process 27]

Add as a process. 27 is an example. Insert the appropriate N+ your PM.cfg requires. This is typically the next available process.

Description

Description of the process.

Process Path

The path where the process lives.

Program Name

The program name. This is typically the name of the .sh file.

Class Name

The class name.

Port

The process port.

Argument

Where you pass any arguments such as configuration (c) or where log files are to be written (l).

In this case, the log files are written to /tango/logs/policy/spcm/spcm‐subscriberprovision‐batch.log.

Run batch provisioning

After you have set up the directories, stored procedures, and scripts needed to execute batch provisioning, you need to provide a CSV file that adheres to a specific format for both subscribers and plans. See CSV file format to ensure your CSV file matches the correct format. Otherwise, you will receive an error in your report file.

This section contains the following sub-sections:

CSV file format
Run the procedure
Report file output
Log file output

CSV file format

The CSV file(s) that are used for batch provisioning must adhere to a certain format. There is a different format for both:

Subscribers
Plans

When ordering the batch files, you should move (or copy) batch files in the order subscriber, plan, subscriber, plan and so on and so forth.

Subscriber CSV format

The subscriber CSV format should contain a subscriber per line with the following fields.

The following fields are required:

msisdn
imsi

Field Description

msisdn

required

The subscriber’s MSISDN in international format.

imsi

required

The subscriber IMSI.

subscriberType

String defining the subscriber type.

prepaid
postpaid
unknown

This field is case sensitive.

subscriberClass

String defining the subscriber class.

This field is case sensitive.

subscriberStatus

String defining the current subscriber status.

active
inactive
barred

This field is case sensitive.

language

String defining the subscribers preferred language. Example: english.

This field is case sensitive.

dpsEnabled

This Boolean flag indicates whether the subscriber is subscribed to DPS for Data.

dpsNotification

This boolean flag indicates whether the subscriber’s DPS notification enabled flag is set or not.

Defaults to value in property spcm.subscriber.dps.notification.enabled.default.

imei

The International Mobile Station Equipment Identity of the subscriber’s handset.

locationZone

The name of the subscriber’s home location zone.

renewalDayOfMonth

The renewal day (if applicable.)

eosNotification

This Boolean flag indicates whether the subscriber’s end-of-session notification message is enabled or not.

Defaults to value in property spcm.subscriber.eos.notification.enabled.default.

paygNotification

This Boolean flag indicates whether the subscriber’s pay-as-you-go notification message is enabled or not.

Defaults to value in property spcm.subscriber.payg.notification.enabled.default.

alternateNotificationMsisdn

This is an alternate MSISDN that will be used as a destination address for notifications if configured.

qoscategoryName

Specifies the Quality of Service category name.

ruleViolationNotification

Specifies the rule violation notification (if applicable).

Example

Subscriber,<msisdn>,<imsi>,<subscriberType>,<subscriberClass>,<subscriberStatus>,<language>,<dpsEnabled>,<dpsNotification>,<imei>,<locationZone>,<renewalDayOfMonth>,<eosNotification>,<paygNotification>,<alternateNotificationMsisdn>,<qoscategoryName>,<ruleViolationNotification>

Plan CSV format

The plan CSV format should contain a plan per line with the following fields.

The following fields are required:

msisdn
plan_name
plan_status

Field Description

msisdn

required

The subscriber’s MSISDN in international format.

imsi

The subscriber IMSI.

plan_name

required

This is a string defining the name of the plan being added for the subscriber. The plan should already exist in the iAX-PCC.

This field is case sensitive.

plan_state

required

This string defines the current state of the plan. The allowed values are:

active
inactive

current_usage

expiry_date_time

This string defines the expiry date for the plan for the specific subscriber. If the parameter is not present then the plan will be unlimited.

plan_info

Identifier used to record the plan instance purchase source.

occurrence_count

Specifies the occurence count for the plan.

max_occurrence_count

Specifies the max occurence count for the plan.

remaining_volume

This numeric value defines the volume of data remaining for the subscriber before migration to the Tango iAX PCC platform.

This value is in bytes.

Example

Plan,<msisdn>,<imsi>,<plan_name>,<plan_state>,<current_usage>,<expiry_date_time>,<plan_info>,<occurrence_count>,<max_occurrence_count>,<remaining_volume>

Run the procedure

Once you have set up batch provisioning, you need to copy or move your CSV file into the /file-input directory. Follow the instructions provided here to run batch provisioning.

Subscribers and plans should only be provisioned on the node with the active database. Do not copy the CSV file onto the node with the backup database.

Prerequisites and requirements

You must have completed the set up batch provisioning.
You must have a CSV file that adheres to the correct format.
You must have full permissions for the SPCM.

Procedure

Move the CSV file(s) into the file-input directory.

mv subscriber-file.csv /tango/data/spcm-provision/file-input/

mv plan-file.csv /tango/data/spcm-provision/file-input/

Check your report file to see if the CSV file was processed successfully or whether you need to troubleshoot.
Check your log file to see if the CSV file was processed successfully or whether you need to troubleshoot.

Report file output

After the batch provisioning job is run, your reports can be found in the /tango/data/spcm-provision/report directory. A sample report with table descriptions are included here to help you get an idea of what to expect when reading a report file.

You can configure a different directory for your reports in the spcm-subscriber-provision-batch.properties file with the spcm.subscriber.provision.batch.report.directory property.

Report file sample

    /tango/data/spcm-provision/report/2MSubs-From-0640000000_1.report

    File Name : 2MSubs-From-0640000000_1.csv (1)
    Start Time : 2013-10-02T14:06:05 (2)
    Read Count : 2000000 (3)
    Write Count : 2000000 (4)
    Skip Count : 0 (5)
    Step Exit Status : COMPLETED (6)
    Batch Status : COMPLETED (7)

Report file field descriptions

Number Field Description Sample

File Name

The name of the file processed.

2MSubs-From-0640000000_1.csv

Start Time

The date and time the file was processed.

This is in ISO 8601 format.

2013-10-02T14:06:05

Read Count

The number of records (rows) read from the file.

2000000

Write Count

The number of records processed and executed.

2000000

Skip Count

The number of records skipped due to error.

0

Step Exit Status

The exit status for the tool itself.

possible values

COMPLETED = completed successfully
COMPLETED WITH SKIPS = completed with some records (rows) missed
FAILED = batch provisioning failed

COMPLETED

Batch Status

The exit status for the batch processing.

possible values

COMPLETED = completed successfully
FAILED = batch provisioning failed

COMPLETED

Log file output

After the batch provisioning job is run, your logs can be found at /tango/logs/policy/spcm/spcm-subscriber-provision-batch.log directory. A sample log file is included here to help you get an idea of what to expect when reading the log file.

You can configure a different directory for your logs in the PM.cfig file by passing the -l argument in the Argument section.

[Process 27]

    Description = SPCM Batch Provision
    Process Path = /tango/bin/spcmBatchProv_pm.sh
    Program Name = spcmBatchProv_pm.sh
    Class Name = spcmBatchProv/0/pmClient/0
    Port = 16050
    Argument = ‐x10100 ‐jprovision ‐c/tango/config/policy/spcm/spcmsubscriber‐
    provision‐batch.properties ‐l/tango/logs/policy/spcm/spcm‐subscriberprovision‐
    batch.log

    Role = OnePerHost
    Logical Host = SPCM Batch Provision Host
    Primary Host = localhost

    Launch Delay = 20
    Initial Response = 60
    Restart Attempts = 3

    Polling Interval = 2
    Outstanding Polls = 3

The file is rolled over at midnight with a date/time stamp.

Log file sample

$ less /tango/logs/policy/spcm/spcm-subscriber-provision-batch.log

2019-12-12 12:40:10,028 INFO  [task-scheduler-10] o.s.integration.file.FileReadingMessageSource : Created message: [GenericMessage [payload=/tango/data/spcm-provision/file-input/CUSTOMER_20191212_0000.csv, headers={id=8946bc3f-26f9-ae13-a2cd-e932f746170e, timestamp=1504262410027}]]

2019-12-12 12:40:10,099 INFO  [task-scheduler-10] o.s.i.file.FileWritingMessageHandler : Failed to move file '/tango/data/spcm-provision/file-input/CUSTOMER_20191212_0000.csv'. Using copy and delete fallback.

2019-12-12 12:40:10,269 INFO  [task-scheduler-10] o.s.b.core.launch.support.SimpleJobLauncher : Job: [FlowJob: [name=batchProvision]] launched with the following parameters: [{file.input=/tango/data/spcm-provision/file-processing/20190901124010059_CUSTOMER_20191212_0000.csv, timestamp=1504262410114}]

2019-12-12 12:40:10,289 INFO  [task-scheduler-10] c.t.s.s.p.b.l.BatchProvisionJobListener : Job start time : [Fri Sep 01 12:40:10 CEST 2019] File name : [/tango/data/spcm-provision/file-processing/20190901124010059_CUSTOMER_20191212_0000.csv]

2019-12-12 12:40:10,306 INFO  [task-scheduler-10] o.s.batch.core.job.SimpleStepHandler : Executing step: [provisionStep]

2019-12-12 12:40:17,222 INFO  [provisionPlan-11] c.t.s.s.p.batch.listeners.DebugChunkListener : Number of chunks completed: count=10, job=batchProvision,step=provisionStep

2019-12-12 12:40:17,925 INFO  [provisionPlan-8] c.t.s.s.p.batch.listeners.DebugChunkListener : Number of chunks completed: count=20, job=batchProvision,step=provisionStep
<snip>

2019-12-12 12:40:22,873 INFO  [provisionPlan-11] c.t.s.s.p.batch.listeners.DebugChunkListener : Number of chunks completed: count=90, job=batchProvision, step=provisionStep

2019-12-12 12:40:22,885 INFO  [provisionPlan-3] c.t.s.s.p.batch.listeners.DebugChunkListener : Number of chunks completed: count=100, job=batchProvision,step=provisionStep

2019-12-12 12:40:22,886 INFO  [task-scheduler-10] c.t.s.s.p.batch.listeners.ReportStepListener : summary StepExecution: id=1003, version=102, name=provisionStep, status=COMPLETED, exitStatus=COMPLETED, readCount=5000, filterCount=0, writeCount=5000 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=101, rollbackCount=0

2019-12-12 12:40:22,895 INFO  [task-scheduler-10] c.t.s.s.p.b.l.BatchProvisionJobListener : Job start time : [Fri Sep 01 12:40:10 CEST 2019], Job end time : [Fri Sep 01 12:40:22 CEST 2019], Duration in seconds : [12], FileName : [/tango/data/spcm-provision/file-processing/20190901124010059_CUSTOMER_20191212_0000.csv]

2019-12-12 12:40:22,896 INFO  [task-scheduler-10] o.s.b.core.launch.support.SimpleJobLauncher : Job: [FlowJob: [name=batchProvision]] completed with the following parameters: [{file.input=/tango/data/spcm-provision/file-processing/20190901124010059_CUSTOMER_20191212_0000.csv, timestamp=1504262410114}] and the following status: [COMPLETED]

Batch provisioning error help

Errors are uncommon when running batch provisioning if the CSV file is formatted correctly, and the tool is set up and configured to run as an application under the control of the process manager. However, if you do encounter an issue with a particular error, it is best to know how to check those errors.

This section does not, of course, contain all of the possible errors that should be checked and is simply a starting point for understanding any errors you might receive while batch provisioning subscribers and plans.

Where do I find batch processing errors?

Errors are sent to the the <csv-file-name>.error file. The .error file uses the same name as the CSV file you provided. An example is shown below with the false (1) errors being shown.

Navigate to this file through your machine’s directory structure or if using linux, you can use the less command. Typically the file is found in the /tango/data/spcm-provision/error/ folder. However, this directory is configureable via the spcm.subscriber.provision.batch.error.directory property in the properties file which is found in the /config folder.

$ less /tango/data/spcm-provision/error/20191212112705015_CUSTOMER_20191212_1000.error

Subscriber,096467134885,176093556911209,UnKnown,OperatorASubscriber,Active,Spanish,false,false,,,1,false,false,, (1)
Plan,096467134885,176093556911209,Plan-Name-A,Active,,2017-03-31T22:00:00,,null,1,
Plan,096467134885,176093556911209,Plan-Name-Other,Active,,2017-03-31T22:00:00,,null,1,

Check the CSV file is formatted correctly

One of the more common errors is that the subscriber (or plan) was not properly defined in the CSV file. If the CSV file is mostly formatted correctly, you might only have one or two subscribers that were not entered correctly.

Check that your CSV file matches the correct CSV file format.

options

Navigate to the file-processed directory of the .csv file that you provided for batch provisioning and check the subscriber.

Run an egrep command with the subscriber MSISDN pointed at the directory where your .csv file resides. An example of this command is shown below.

$ egrep 096467134885 /tango/data/spcm-provision/file-processed/20191212112705015_CUSTOMER_20191212_1000.csv

Subscriber,096467134885,176093556911209,Unknown,OperatorASubscriber,active,Spanish,0,0,,,1,0,0,
Plan,096467134885,176093556911209,Plan-Name-A,active,,,
Plan,096467134885,176093556911209,Plan-Name-Other,active,,,

Typically the file-processed directory is set at /tango/data/spcm-provision/file-processed. However, this is configurable in the properties file so check your file to make sure you are looking in the right place.

Check if the subscriber is already in the SPCM

If you receive an error for a particular subscriber in batch provisioning, it is possible that the subscriber already exists in the SPCM. In this case, use the SPCM API to check and see if the subscriber already exists. If the subscriber does exist, go on to see if a plan was purchased before you attempted the batch provisioning. If a plan was purchased (see purchaseTimestamp parameter in the JSON response) before batch provisioning occurred. This is why the subscriber threw an error.

See the Get subscriber plans and Get subscriber info API operations for instructions on how to do this.