Archive for the ‘devops’ Category

The Poor Man’s Job Queue

Not all software development projects are treated the same, some have access to modern tech Virtual Private Servers (VPS), Zend Server (http://www.zend.com/products/server/), Memcached, Gearman and all the other goodies I can only dream of. You have a box with LAMP, and you cannot install anything else.

This is an example of how we got around a limitation, using available tools. Problem: I have a list of tasks to execute within my application, however I need to ensure that the tasks are executed and completed, but some are more important than others, and the execution may slow down the performance of the box we are running on. Well in this case we were loading 6 different types of XML files which were FTPed into a location on the box regularly, every 35 minutes and had to be loaded in a specific order. This was further complicated by the fact that we had to reload historical data in case of issues (1 weeks worth of uploads ~ 2100 files) without interrupting the current loading processes.

The approach used the following components:

a) Job Queue – based on the Zend Server Job Queue but simplified for our needs (see data model of tables below)

Job Queue Data Model

Job Queue Data Model

b) Queue Loader Script – loads the jobs into the job queue by scanning the location containing the files to be loaded and adds the files to the queue (since the queue is a database table, duplicates are discarded without errors) This keeps this file simple and honest

c) Job Executor Script – reads a message from a queue, reads the message body which contains the file name to be processed, could be made more complex

d) Queue Loader Cron Job – calls the Queue Loader Script to add new files to the queue

e) Job Executor Cron Job – calls the job executor script. This job has no effect if a lock file exists, and is not expired which means the script is valid and running. However if the lock file is expired, it means that the process crashed, so the lock file is deleted, a new process is started with a lock file. Basically this keeps the job executor script running indefinitely as long as there are messages to process. 

Please feel free to leave a comment on what your experiences are with similar problems. 

Launch of Sim Card Registration by Uganda Communications Commission – March 5, 2012

It is a Monday morning, and 7:00am as requested I am at the Sheraton Kampala for the launch of Uganda Communications Commission (UCC) official launch of the sim card registration which requires all mobile phone users to register their sim cards with the Telcos. The telecoms are setting up registration tents outside so I think I will register my 4 (yes four) sim cards today and get it over and done with.

I already have my finger prints and photos taken for my drivers license, have my details also taken by two telecoms (MTN and UTL) for their mobile money services. Not forgetting I have to register with the other two telecoms for their mobile money services too 🙂

Today is the stakeholder launch and the public launch will be on March 24, 2012 at Nakivubo stadium. In attendance are the top guns of the telecoms, Security Minister and Inspector General of Police, Executive Director NITAU, members of parliament so it seems like the project has political buy-in. The social and technical challenges well are still yonder.

The driving factor for the sim registration is to curb the wave of crime perpetuated by explosion of mobile usage in Africa over the last decade based on the numbers from the ITU. This explains why the advertising theme for the sim card registration is “Make Communications Safe” and the messages are: no more hiding by bullies and conmen/conwomen, sim people have bad intentions. However this begs the question “Are there no positive messages to show how beneficial it is to register the sim cards?”

Critical issues that I am looking to see mentioned better still addressed:

1. Is the information to be synchronized across the different telecoms? – Answer: Each telco is charged with registering the subscribers within their network and securely storing the information within their system. This raises a question of interoperability between the information stored by the different operators on their systems.

2. Is the sim card registration also to be synced with mobile money registration too, or are they to be kept separate?

3. How are privacy implications to be addressed? Answer – This is the responsibility of the different telecoms overseen by UCC. The government is reassuring the public that the telecom providers will protect this information. With no details this is out in the wild.

4. How is this process to be scaled to the 40% sim card users in the rural populations who actually do not have any form of registration?

5. How will corporate registrations of sim cards to be handled?

6. How will this link up with all the other registration systems, National ID, Drivers license, Credit Reference Bureau, and any new ones that will be thought up too …?

7. How will verification of the registration information be done, do we assume that all who register are using their real names and information? Answer – the government will no tolerate any such activities

8. Who owns the registration information – the telecoms, UCC, Government of Uganda, the registrants? Answer – The information seems to be owned by the telecoms who capture the subscriber information.

I have registered my 4 sim cards on all services and here is my take on the operational challenges so far:

1. This is a chance for the service providers to sell their mobile money services, since the sim card registration is invariably mobile money registration too. This puts pressure on the incumbent MTN Uganda which has the largest foot print

2. The telcoms are ill prepared for the logistical nightmare that the sim card registration calls for, and will put pressure on their earnings for the next 2 years. We were only about 100 people at the stakeholder launch, but it took almost 20min at each providers stall. Mulitply this 10,000 fold and you get the picture with only 10% of estimated subscribers covered. Lessons from credit reference bureau service roll-out planning should have been used as it was done to over 500,000 bank account holders and was tied to regulatory compliance by financial institutions.

3. The duplication of efforts is daunting. My opinion is that UCC should have forced the telcoms to come together and carry out this registration as a block for it to be successful.

4. Information privacy is still a major issue which has not been addressed, we are being told to trust the telecoms.

5. There is no verification of information, and it is easy to get and use forged credentials for sim card registration which becomes official. This could have been simplified if the registration has been done by a block of telecoms.

On a parting note, as I always have them Isiah Katumwa’s saxophone playing is off the hook, what talent…

MTN Uganda, Mobile Money and Operations Issues

I am not one to rant and rave but I seem to have been pushed over the edge this morning, but a large Telco service which leaves a lot to be desired yet despite being innovative seem to be leading more and more wastage in terms of time which would be used for more productive pursuits.

The service is Mobile Money, currently being hailed as Africa’s savior in terms of providing financial services to the millions of unbanked populace. Everybody knows that mobile telcom services in Africa have been very successful and are growing by leaps and bounds due to the infrastructure issues associated with fixed line laying, operation and maintenance. Couple the cost of handsets, $10 Nokias are available with a battery that can last 5 to 7 days, oh yes, coupled with SMS has lead to mHeath, mEducation initiatives being developed.

Mobile money has been a core driver of mobile service usage in the last few years coz it makes it easy to move money without the hassles of banks (line up, service fees) and with the licensing of thousands of agents (there are now more agents than bars and supermarkets and groceries combined), means that getting access to money is as easy as moving to your local grocery store.

However MTN Uganda (http://mtn.co.ug/) is a market leader in Uganda and currently holds the market leadership position, I would put it at over 70% but I can be corrected, with the greatest reach within the country. The service is estimated to transact about UGX 5bn ($2.2m at current rates) per day which is quite high considering averge transaction values are in the $10 – $100 range.

Anyway their success is maybe their undoing, because despite the phenominal growth, the service is even worse the electrictity availabiltity with the platform having an average uptime of 50% during normal working hours, after a 45 day downtime during November 1, 2011 – December 15, 2011 (which started as an upgrade then later turned into an outage).

From my software engineering background I am still baffled at why this continuously happens to one of the largest telco providers due to the established DevOps (http://devops.com/ and http://en.wikipedia.org/wiki/DevOps) practices: what are the possible solutions or approaches:

  1. High Transaction Volumes
    • Hardware – buy more hardware throw more power at the problem
    • Software – not scalable then run a cluster of boxes across the switches, load balance the sessions this problem is available even with HTTP
  2. Interface Operations – In database speak we usually state separate writes from reads. Separate balance checking (reads) from  withdrawals and deposits (writes) into separate distinct applications behind the interface. Use Queues, Gearman to ensure that the transactions are completed. Have the reads, balance checks run off slaves in the clusters …
  3. Notifications – SMS  Messages are good, for delivery but ensure they are sent and delivered. Queue the notifications so that they are always sent
  4. Provide options to execute transactions – provide a web interface for clients and agents. This opens up new revenue and agent opportunities since Internet cafe owners can also provide services from their interfaces. This is just an alternate way to access the service
  5. Be open to the public to lower the expectations – provide updates on service outages so that users do not just keep trying and only finding out from many failed trials. Failed transactions have been identified a known cause of application load spikes
  6. Reduce the number of available services and offload some services to other channels
  7. Use opensource software it has been proven to scale – or maybe some newer versions of your software applications
  8. New – Provide APIs so that developers can provide custom solutions to offload processing off your core system (switchboard)

These are just quick thoughts but they should be sufficient to start the discussion … not only rant and rave but also provide some concrete solutions

%d bloggers like this: