Archive for the ‘devops’ Category

Alternate Approach to Legal Independent Election Tallying

The Uganda elections are more or less over with less than 6 hours for the Uganda Electoral Commission (EC) to announce the results for the presidential elections.

Given all the time on our hands, with no social media, the team at Styx Technology Group designed the following alternative approach to independent electoral vote tallying for future elections that provides inbuilt mechanisms for audit and verification of results.

The primary data sources for the process are:

  1. Official EC list of polling stations and voters per polling station
  2. Photos of the signed election tally sheets from each polling station. To ensure that the photos are not tampered with and provide an audit trail:
    • Each photograph has to be taken with information on the camera, the GPS coordinates of where the photo was taken, date and time when the photo was taken which is available in many cameras that share it using the Exchangeable Image File Format (EXIF)
    • Two separate photos of the tally sheets have to be taken by different cameras
    • The cameras taking equipment may be registered beforehand to provide validation of the source of the information
    • The signatures of the returning officers and stamp must be clear and visible in the photo

The architecture for the technology solution is as follows:

  1. Web based solution accessible via any browser. Due to poor Internet connectivity in many areas of the country, an Android app would be provided to assist in data collection, then data sent once the user gets into an area with Internet.
  2. The field officers who capture the photos would also be provided with an option of entering the candidate vote tallies.
  3. In the tallying center, candidate vote tallies are entered from the photos received and vote tallies entered by data clerks. In order to reduce errors the following approach would be used:
    • The clerks are randomly assigned photos as they come in
    • The tally for a station must be entered correctly by two separate data entry clerks, then approved by a supervisor. This process is formally called the two-pass verification method or double data entry.
  4. All correctly entered data is shared with the rest of the world for download and analysis.

This system is mission-critical having to be available for the entire vote counting period of 48 hours,  so the architecture includes the following paths for data collection:

  1.  Multiple access IP addresses and domains for the website in case some are blocked off
  2. Any data collected via the Android app can be sent via email to a dedicated tallying center address. To ensure that only data from the app is received and not changed in transit, encryption is used.

The inspiration came from a quote by Ghandi “Be the change you wish to see in the world”, disproving the myth that there is no local capability to design and implement such solutions and most of all that such solutions have to be complex.

Looking forward to hearing your thoughts and suggestions…

Trunk or Branch based development

An interesting discussion that I had with my team mates over the last few days, whether we should create branches then merge later or keep working on the trunk within our Git based version control process. As the team is small, we are in the same premises but different locations, we agreed to move to work exclusively on the mainline for the following reasons:

  1. Reduce the amount of work having to remember which branches are active, so branches are an exception rather than the rule
  2. Adding practices like a CI pipeline (that’s additional work for all of us to setup) will provide a needed stability in the long-run as some of the projects are expected to be long running
  3. Working on the main line forces us to talk to each other, rather than IM away, so design decisions are shared across the team
  4. Branches discourage refactoring mostly due to the pain of merging refactored changes, and the fact that not everyone can benefit from the refactoring as soon as its completed – thanks to Twitter – Chris Ford

We used the following resources as research:

1. Martin Fowler – Feature Branch – http://martinfowler.com/bliki/FeatureBranch.html also talking about Feature Toggles – http://martinfowler.com/bliki/FeatureToggle.html

2. Apologists Defense of Trunk based development – http://www.tuesdaydeveloper.com/2015/05/an-apologists-defense-of-trunk-based-development/

3. What is Trunk Based Development http://paulhammant.com/2013/04/05/what-is-trunk-based-development/

4. Shades of Trunk based development – http://paulhammant.com/2014/09/29/shades-of-trunk-based-development/

What do you use with your team and why?

Backup vs Murphy’s Law – My Take

This blog post has been inspired by events, not to me, but to a colleague of mine. Our discussion, was interesting, funny and insightful, but I thought hey why not share my experiences and what I use to teach others how to live a safer, “backup” driven life.

I live on Murphy’s law, for those of you who do not know what it is, “If anything can go wrong, it will” and with extensions “At the most inopportune time” and my favorite “it will be all your fault, and everyone will know it”. Basically it means that whatever you do, the closer you get to success the greater the chance of failing terribly, at the last moment. Therefore successful people are those who plan for the inevitable or develop habits which reduce this possibility.

My first experience with disaster came as part of my first job, an internship with an IT company many years ago. I was given the task of entering 2 years worth of petty cash transactions into an Excel spreadsheet, which I was saving on a floppy drive (yes we used them at that time). Anyway I worked on the file for 3 weeks, and then just as I was done (at the most inopportune moment), the floppy drive crashed, yes beyond repair, it was a mechanical fault. Since then I have never lost another file again, and I would like to share what I do, it may be a little extreme, but hey I am paranoid.

Maintain Multiple File Versions

Seems simple, but when do you decide to make a new version. My rules for creating versions are simple:
– At each major change in structure or layout or content
– Each day

So many files, oh yes, I create an “Archive” directory in each folder I work in so that I move the older files in there, till I next need them. At times never. Also I name all my files the same way FileNameDDMMMYY for example BackupBlogPost11Jul12.txt (that I am working in now). Tomorrow I will create a new version of the file BackupBlogPost12Jul12.txt.

Email – Copy Yourself

Strange, why not keep it in Sent Items, no way, I always copy myself on each and every communication. Some people would argue that “Sent Items” in MS Outlook or web mail is good enough, but I think not.

The advantages are:
– Since I apply rules to file my emails and Labels in Gmail, I only focus on the Incoming mail and can empty Sent Items and Drafts as often as I need to.
– I can find all threads and timestamps for when I sent the communications (I had to retrieve a contract ammendment sent 3 years before in one case)
– Thanks to Gmail I do not have to delete any emails so its a permanent record

Backup

Yeap the same old story, its like washing my hands, nope really, but backups are over-rated and always forgotten because they are somewhat of a dark art. They have to be simple, reliable and automated. For personal use my backup options are:
– Dropbox seems like I have over 4GB now
– Google Docs becoming more interesting since I can edit the files using Docs and Spreadsheets
– Online Tools that suit the type of file at hand

File Storage – Magnetic vs Flash

I do not trust flash drives, and I believe Magnetic drives (HDDs) are more resilient but I am yet to have my mind corrected.

What do you do in your case? Please share your thoughts in comments.

MeetMTN – MTN Uganda Executive Management Town Hall Meeting

Well 2012 is upon us, and the incubmbent US President Barack Obama, has started his battle for a second term in office, so that he can complete the change that he promised the American people. The battle for his first term was epic, first against Ms. Hillary Clinton, which intially changed from a David vs Goliath story to a battle of titans that he won. The one against the Republican was a massacre as Mr. John Mccain shot himself in the foot and head one campaign after another.

Why all this about President Obama, well one of his campaign tactics was town hall style meetings in different towns, and constitencies in order to connect with the people in the ground, and hear what they have to say directly.

Teleport straight to Uganda the Pearl of Africa, home to great beauty among other things, as well as to multi-national corporations which are running a large chunk of the core economy activities. Well these corporations have been facing a backlash with regard to service delivery over the last few years (which was terrible), the wrath of most was directed to UMEME the national power distributor due to the incessant power shortages which sort of brought the economy to its knees in 2011. At the height of the backlash, UMEME started to reach out to the populace to inform and educate the general public who did not understand their roles, challenges and strategies for attacking the rampant power problem. One of these events was a Tea Party for social media enthusiasts – bloggers and tweeps covered here https://ssmusoke.wordpress.com/2011/12/09/umeme-tea-party-quick-and-dirty-review-and-perspective/.

Well MTN Uganda, the largest telco by market share, numbers and revenues, has also reached out using a similar strategy this time called MeetMTN, where tweeps and bloggers were invited to meet the MTN executive management to discuss issues pertaining to service delivery and what is being done to work around the known issues.

Here is a summary of the major issues raised during the meeting, and responses from MTN executives who attended:

1. Provision of an API for Mobile Money and development of a partner ecosystem – information on when an API will be provided for mobile money as the largest provider and one with the largest reach. From the recently completed startup weekend, 12 of the 13 startup ideas depend on Mobile Money for payments. However no details of the availability of an API for 3rd parties were given citing a need not to commit to predefined dates. .The API is coming soon

2. Staff Attrition – there are rumours in the grapevine that there is a higher than normal attrition rate of talent over the last quarter, and whether this is one of the root causes of the recent outages and lowered quality of service delivery. The CEO mentioned that MTN leverages its regional strengths, but focuses on developing the local talent within the countries that MTN operates.

3. Service Health Dashboard – whether its possible to provide a public dashboard on the health of different services, inspired by Google approach. Also whether it was possible to spread information on service slowdowns and outages through different channels, Facebook/Twitter/Website/SMS, Radio/TV and customer service centers for extended service outages.

4. Lottery Games – there was a game which involved amassing points by answering questions for a chance to win UGX 10,000 (~US$4,000), and one of the attendees played the game using his family resources, and his wife almost divorced him. He was wondering how many points were needed to win the prizes so that he would be contented. The executives thanked him for playing and wished him better luck next time.

5. Unsolicited, Spam and Junk SMS messages plus hidden charges from content providers licensed by MTN and the company itself – There are many SMS messages for caller ring back tones, both from MTN and licensed content providers both as spam and also unknown activations which charge user numbers. We were informed of a blacklist code to enter I seem to have forgotten it, but this is not publizied … similar to US “Do Not Call Registry”

6. Does MTN Sell Customer numbers to content providers – MTN does not sell numbers to content providers, however these numbers are collected as part of normal usage for example promotions etc which may be done by friends, relatives. Ms. Maureen Agena of Text-to-Change also provided her organization’s experience with collecting targetted numbers which corroborates the information and policies from the executive.

7. Postpaid Service upgrade – customers on post paid services cannot upgrade data bundles in case they run out on a pay-as-you go service once the post paid bundle runs out. The executive informed us that the billing platform for all services is converged therefore this should be cleared in the very near future.

8. Hoax Promotions – there are cases when a promotion sale of phones or devices is announced, but they are not available because they were gobbled up by MTN staff. Management promised to look into this

9. IPV6 Migration – internal testing is in progress, but roll out to all customers is not confirmed until the services are confirmed as stable

10. SME Pricing – there seem to be no plans geared towards the SME sector as the current plans are either consumer (too expensive for SME at retail rates), or Enterprise (capex and opex costs too high for SME)

UPDATES – from Reader Feedback:

11. Simcard registration process and charges – there are claims of some agents charging for sim-card registrations, bu there is no charge for sim-card registrations. Also MTN intends to use this exercise as a “Know Your Customer” exercise and to deal with the spam/junk SMS message problem highlighted above.

12. Mobile Hotspots – the internet is not usable in a few parts of the country, and it would be better if there were hotspots at least in the urban areas, so that customers do not need dongles but just connect, and pay for usage. The technical team informed us that this was something that they were considering.

13. Internet Bundles using airtime once they expire without approval – currently once the Internet bundles expire the user’s airtime is charged which is the default option. There are ways around this and MTN intends to educate its customers on how to use these options

14. Slow Speed of the Website – the website is too slow, actually one of the slowest in Uganda, and the website self service features do not work. The website is currently being upgraded for speed and features, and a new one will be unveiled soon (BTW this is very embarrassing for multi-national telcom company).

Overall the interactions were excellent, and the available MTN staff were knowledgeable of subject matter they were discussing, which left a ray of hope that finally the behemoth is listening to its customers. Obviously the gift packs at the end made the trip worthwhile too. In closing I would like to say again as I did on Twitter:

“The only way @MTNUGANDACARE can support #startups and #enterprenurs is to provide a #mobilemoney #API #MeetMTN”

If I missed anything, just let me know …

The Poor Man’s Job Queue

Not all software development projects are treated the same, some have access to modern tech Virtual Private Servers (VPS), Zend Server (http://www.zend.com/products/server/), Memcached, Gearman and all the other goodies I can only dream of. You have a box with LAMP, and you cannot install anything else.

This is an example of how we got around a limitation, using available tools. Problem: I have a list of tasks to execute within my application, however I need to ensure that the tasks are executed and completed, but some are more important than others, and the execution may slow down the performance of the box we are running on. Well in this case we were loading 6 different types of XML files which were FTPed into a location on the box regularly, every 35 minutes and had to be loaded in a specific order. This was further complicated by the fact that we had to reload historical data in case of issues (1 weeks worth of uploads ~ 2100 files) without interrupting the current loading processes.

The approach used the following components:

a) Job Queue – based on the Zend Server Job Queue but simplified for our needs (see data model of tables below)

Job Queue Data Model

Job Queue Data Model

b) Queue Loader Script – loads the jobs into the job queue by scanning the location containing the files to be loaded and adds the files to the queue (since the queue is a database table, duplicates are discarded without errors) This keeps this file simple and honest

c) Job Executor Script – reads a message from a queue, reads the message body which contains the file name to be processed, could be made more complex

d) Queue Loader Cron Job – calls the Queue Loader Script to add new files to the queue

e) Job Executor Cron Job – calls the job executor script. This job has no effect if a lock file exists, and is not expired which means the script is valid and running. However if the lock file is expired, it means that the process crashed, so the lock file is deleted, a new process is started with a lock file. Basically this keeps the job executor script running indefinitely as long as there are messages to process. 

Please feel free to leave a comment on what your experiences are with similar problems. 

Launch of Sim Card Registration by Uganda Communications Commission – March 5, 2012

It is a Monday morning, and 7:00am as requested I am at the Sheraton Kampala for the launch of Uganda Communications Commission (UCC) official launch of the sim card registration which requires all mobile phone users to register their sim cards with the Telcos. The telecoms are setting up registration tents outside so I think I will register my 4 (yes four) sim cards today and get it over and done with.

I already have my finger prints and photos taken for my drivers license, have my details also taken by two telecoms (MTN and UTL) for their mobile money services. Not forgetting I have to register with the other two telecoms for their mobile money services too 🙂

Today is the stakeholder launch and the public launch will be on March 24, 2012 at Nakivubo stadium. In attendance are the top guns of the telecoms, Security Minister and Inspector General of Police, Executive Director NITAU, members of parliament so it seems like the project has political buy-in. The social and technical challenges well are still yonder.

The driving factor for the sim registration is to curb the wave of crime perpetuated by explosion of mobile usage in Africa over the last decade based on the numbers from the ITU. This explains why the advertising theme for the sim card registration is “Make Communications Safe” and the messages are: no more hiding by bullies and conmen/conwomen, sim people have bad intentions. However this begs the question “Are there no positive messages to show how beneficial it is to register the sim cards?”

Critical issues that I am looking to see mentioned better still addressed:

1. Is the information to be synchronized across the different telecoms? – Answer: Each telco is charged with registering the subscribers within their network and securely storing the information within their system. This raises a question of interoperability between the information stored by the different operators on their systems.

2. Is the sim card registration also to be synced with mobile money registration too, or are they to be kept separate?

3. How are privacy implications to be addressed? Answer – This is the responsibility of the different telecoms overseen by UCC. The government is reassuring the public that the telecom providers will protect this information. With no details this is out in the wild.

4. How is this process to be scaled to the 40% sim card users in the rural populations who actually do not have any form of registration?

5. How will corporate registrations of sim cards to be handled?

6. How will this link up with all the other registration systems, National ID, Drivers license, Credit Reference Bureau, and any new ones that will be thought up too …?

7. How will verification of the registration information be done, do we assume that all who register are using their real names and information? Answer – the government will no tolerate any such activities

8. Who owns the registration information – the telecoms, UCC, Government of Uganda, the registrants? Answer – The information seems to be owned by the telecoms who capture the subscriber information.

I have registered my 4 sim cards on all services and here is my take on the operational challenges so far:

1. This is a chance for the service providers to sell their mobile money services, since the sim card registration is invariably mobile money registration too. This puts pressure on the incumbent MTN Uganda which has the largest foot print

2. The telcoms are ill prepared for the logistical nightmare that the sim card registration calls for, and will put pressure on their earnings for the next 2 years. We were only about 100 people at the stakeholder launch, but it took almost 20min at each providers stall. Mulitply this 10,000 fold and you get the picture with only 10% of estimated subscribers covered. Lessons from credit reference bureau service roll-out planning should have been used as it was done to over 500,000 bank account holders and was tied to regulatory compliance by financial institutions.

3. The duplication of efforts is daunting. My opinion is that UCC should have forced the telcoms to come together and carry out this registration as a block for it to be successful.

4. Information privacy is still a major issue which has not been addressed, we are being told to trust the telecoms.

5. There is no verification of information, and it is easy to get and use forged credentials for sim card registration which becomes official. This could have been simplified if the registration has been done by a block of telecoms.

On a parting note, as I always have them Isiah Katumwa’s saxophone playing is off the hook, what talent…

MTN Uganda, Mobile Money and Operations Issues

I am not one to rant and rave but I seem to have been pushed over the edge this morning, but a large Telco service which leaves a lot to be desired yet despite being innovative seem to be leading more and more wastage in terms of time which would be used for more productive pursuits.

The service is Mobile Money, currently being hailed as Africa’s savior in terms of providing financial services to the millions of unbanked populace. Everybody knows that mobile telcom services in Africa have been very successful and are growing by leaps and bounds due to the infrastructure issues associated with fixed line laying, operation and maintenance. Couple the cost of handsets, $10 Nokias are available with a battery that can last 5 to 7 days, oh yes, coupled with SMS has lead to mHeath, mEducation initiatives being developed.

Mobile money has been a core driver of mobile service usage in the last few years coz it makes it easy to move money without the hassles of banks (line up, service fees) and with the licensing of thousands of agents (there are now more agents than bars and supermarkets and groceries combined), means that getting access to money is as easy as moving to your local grocery store.

However MTN Uganda (http://mtn.co.ug/) is a market leader in Uganda and currently holds the market leadership position, I would put it at over 70% but I can be corrected, with the greatest reach within the country. The service is estimated to transact about UGX 5bn ($2.2m at current rates) per day which is quite high considering averge transaction values are in the $10 – $100 range.

Anyway their success is maybe their undoing, because despite the phenominal growth, the service is even worse the electrictity availabiltity with the platform having an average uptime of 50% during normal working hours, after a 45 day downtime during November 1, 2011 – December 15, 2011 (which started as an upgrade then later turned into an outage).

From my software engineering background I am still baffled at why this continuously happens to one of the largest telco providers due to the established DevOps (http://devops.com/ and http://en.wikipedia.org/wiki/DevOps) practices: what are the possible solutions or approaches:

  1. High Transaction Volumes
    • Hardware – buy more hardware throw more power at the problem
    • Software – not scalable then run a cluster of boxes across the switches, load balance the sessions this problem is available even with HTTP
  2. Interface Operations – In database speak we usually state separate writes from reads. Separate balance checking (reads) from  withdrawals and deposits (writes) into separate distinct applications behind the interface. Use Queues, Gearman to ensure that the transactions are completed. Have the reads, balance checks run off slaves in the clusters …
  3. Notifications – SMS  Messages are good, for delivery but ensure they are sent and delivered. Queue the notifications so that they are always sent
  4. Provide options to execute transactions – provide a web interface for clients and agents. This opens up new revenue and agent opportunities since Internet cafe owners can also provide services from their interfaces. This is just an alternate way to access the service
  5. Be open to the public to lower the expectations – provide updates on service outages so that users do not just keep trying and only finding out from many failed trials. Failed transactions have been identified a known cause of application load spikes
  6. Reduce the number of available services and offload some services to other channels
  7. Use opensource software it has been proven to scale – or maybe some newer versions of your software applications
  8. New – Provide APIs so that developers can provide custom solutions to offload processing off your core system (switchboard)

These are just quick thoughts but they should be sufficient to start the discussion … not only rant and rave but also provide some concrete solutions

%d bloggers like this: