Archive for the ‘database’ Category

Doctrine 2 – Day 2 – Model Validation using Symfony Validator Service in Zend Framework

It is just Day 2 of my experiences in the trenches, regular work, had kept me from this but I managed to get some time to keep digging. As a followup to my Doctrine 2 – Day 1 – Commentary from the Trenches. The models are up and configured, and the unit testing is setup following steps from Michelangelo van Dam’s presentation (

One of the major process that we are implementing with this migration is detailed unit testing which was tougher with the Doctrine1. With the unit test infrastructure setup, the next item on the agenda is model validation. From previous experience (before Doctrine 1) and with Doctrine1 it is critical to be able to specify validation rules using annotations without having to write PHP if statements. Being a ZF person, the first step to look was the ZF validator classes. While they seem to be well integrated with the forms, they would prove to be too verbose to use for validation of models, since I also need to be able to specify multiple validators per column, and this would not cut it for me.

Next stop was Symfony2 validator service ( provides validators with annotations support. So that was the easy part, the hard part was yet to come integration. The integration followed the steps below:

a) Add Symfony Validator service to library folder, easy, just download a package from

b) Register the Symfony Validator annotations – this is where I had problems (more later)

c) Add the Symfony Validators to the model properties

d) Add validation code which needed a validate() method in the base class from which all entities are derived, which requires @ORM\HasLifecycleCallbacks (so that the model can hook into the lifecyle call back models) @ORMPrePersist and @ORM\PreUpdate for the validate method to ensure its called before the models are saved (first time) or updated. More details on the Doctrine annotations can be found at

So what problems did I face.

1. From the Doctrine documentation, you use the annotations directly, for example @Id, however experience has shown that you need to namespace and alias them see note from Symfony integration . So I had to change all the Doctrine annotations to use @ORM namespace

2. The default annotation driver only supports a single namespace so you will need to update as per the pastebin below

Now I am a happy camper, got my models working using Symfony validations, we only have to write code for custom validations which happens only about 20 – 30% of the time.

As a parting shot, the Symfony team and community have done a great job for PHP, why because they provide standalone components (similar to Zend Framework), but each of their components can be used without the rest of the framework. As I was investigating the validator usage and issues, I found a thread where Fabien Potencier and team were discussing annotation support in Symfony. However they also noted that Doctrine Commons had better support, so they stopped the work on Symfony annotation support and just used the services of the Doctrine team. This is how all software development should be done, and is a torch to the rest of us. I am a convert, and happy to be a proud member of the PHP community.

Update May 8, 2012

I had promised to provide some sample snippets of what I am using for the integration with Bisna integration and Symfony validation that I ended up using so here we go The files are as follows:

a) application.ini – there is nothing special here from Bisna. Includes the cache configuration, prod/staging/dev/test environments all of which inherit from production

b) index.php – from the public folder or htdocs – this may not be perfect but it works and am looking for ways to simplify it

c) Bootstrap – this is the file we use, highlight:

– Storing the entity manager instance in the Zend_Registry, we have a utility method which loads it from the registry and another which also provides a connection from the entity manager so its fully encapsulated

– intialization of the Zend_DB adapter, we need this since we are using the Zend_Session to save the sessions in the database

– the last config is for other resources we use. We have a dependency on the Zend_Registry class as it hides a lot of complexity

d) Document.php – a sample model class

– it extends BaseEntity which provides automatic getters and setters through the __call method, some required fields like id, datecreated, last  update date and last updated by (not all models extend this class only those which need those auditing fields)

– Why do we have getCategoryID(), setCategoryID() for the category property instead of mapping the categoryid field from the database see the next post in the series

– Unlike the Doctrine2 defaults we do not use tablename_id but rather tablenameid for the foreign keys so we have define them in each relationship.

Please let me know what I can do to make this any clearer, thanks for reading


State of PHP Feb 2012 – Symfony 2, Zend Framework 2, IDE Support, MySQL

There is just too much going on so I thought I should put these thoughts down so that I do not lose my mind from the excitement and anticipation that comes from things moving very fast and innovation being spured at a pace which is mind boggling.


In my opinion the stabilization of PHP 5.3 which introduced namespaces (similar to Java Packages) was the first shot across the bow, and I expect 5.4 with traits to bring even more reusability. However our hosting providers are slow to upgrade most are still running PHP 5.2, without general availability for 5.3 just starting to appear


There are two main frameworks in the bull pen Symfony 2 ( and Zend Framework 2 ( currently in beta. However they are as different as can be. While Symfony 2 has been refactored from a full blown framework to a bunch of core reusable components, its moving towards ZF2, in that its not trying to be all MVC, forms etc, but is rather a set of building blocks to provide web application framework. So good is it that Fabien Potencier the lead developer has a series on how to build your own framework on top of Symfony at

ZF2 on the other hand is a bunch of reusable components which you can mix and match with other frameworks which also provides an MVC, ORM etc.

If you have been in the Java world, Symfony is more like Spring Source () while Zend Framework is Apache Jakarta (

Interesting tweet that sums it all up:
Every time a developer complains about php, the #Symfony2 community creates another great, namespaced, decoupled and reusable component

IDE Support

Zend Studio 9 was let loose at ZendCon, Oracle released Netbeans 7 (which supports ZF1, and Symfony), Eclipse PDT is chugging along, PHP Storm from Jetbrains creating  a storm with others  Vim, Notepad++, Sublime Text basically all you need to do is pick your poison.


Github ( is the new Facebook and LinkedIn for developers, where all the action is happening for all the major PHP communities, you had better be there. Forking and making changes is as easy as a button press. I am yet to push my first commit to a project but I am getting there.


The best know web database is getting losta love from Oracle, and also the community led by Percona, SkySQL, Monty DB, and many others. Where it seemed lost with lost of FUD during the $1bn Sun takeover by Oracle, it seems to just be gaining steam (at least from my viewpoint as a consumer, developer and user). Now at 5.5, with 5.6 starting to appear over the horizon …

While there is a barrage of NoSQL solutions, Hadoop leading the charge, MySQL provides a NoSQL interface via HandlerSocket ( and MySQL Cluster (

Supporting Infrastructure

This is where all the fun happens, who is using the language and what are the trends. Here is a snapshot:

  1. Content Management Systems – WordPress is getting lots of love, while Joomla now at 2.5 supports databases other than MySQL. Drupal is adopting Symfony 2 as its core framework while Magento runs off a modified ZF1 kernel, while XOOPS ( is being rewritten to use ZF1 or two
  2. Template Engines – an area where battles are found won, and lost. Smarty now at 3.0 (, has to battle Twig (  which is surging due to its usage in Symfony 2
  3. Web services – JSON, XML, AjaX, Rest Interfaces are no longer buzz words they are the norm with native support being baked into PHP
  4. Object Relational Mappers (ORMS) are one of the core developer architectural choices, in PHP we have ZF Table (Table Gateway), Doctrine 1 (Active Record), Doctrine 2 (Data Mapper), Propel (not used) in addition to plain old PDO all of which provide rich choices. Interesting is that Doctrine 2 provides an mapping for MongoDB which shows its versatility similar to Hibernate for Java
  5. Continuous Integration – with Hudson and Jenkins, and now Travis which is also hooked into Github provides a great way to continuously monitor the quality of your development
  6. Quality Control – PHP Unit has always been there but now we have mess detectors, and other metrics (do not make sense of them yet), but whatever you want to measure there is probably a tool for it.
  7. UI Toolkits – JQuery took the world by storm with JQuery UI providing a set of theme-able components, but we have HTML5 boiler plate and Twitter Bootstrap which are simplifying the cross browser issues we developers face.
  8. Performance – always a PHP problem but it has improved through 5.3 to 5.4, and other techniques like OP Caches are becoming more maintream, Memcache is pre-built into Apache …

Exciting times to be a PHP web developer oopsss we are not web developers we are Software Mechanics (

MTN Uganda, Mobile Money and Operations Issues

I am not one to rant and rave but I seem to have been pushed over the edge this morning, but a large Telco service which leaves a lot to be desired yet despite being innovative seem to be leading more and more wastage in terms of time which would be used for more productive pursuits.

The service is Mobile Money, currently being hailed as Africa’s savior in terms of providing financial services to the millions of unbanked populace. Everybody knows that mobile telcom services in Africa have been very successful and are growing by leaps and bounds due to the infrastructure issues associated with fixed line laying, operation and maintenance. Couple the cost of handsets, $10 Nokias are available with a battery that can last 5 to 7 days, oh yes, coupled with SMS has lead to mHeath, mEducation initiatives being developed.

Mobile money has been a core driver of mobile service usage in the last few years coz it makes it easy to move money without the hassles of banks (line up, service fees) and with the licensing of thousands of agents (there are now more agents than bars and supermarkets and groceries combined), means that getting access to money is as easy as moving to your local grocery store.

However MTN Uganda ( is a market leader in Uganda and currently holds the market leadership position, I would put it at over 70% but I can be corrected, with the greatest reach within the country. The service is estimated to transact about UGX 5bn ($2.2m at current rates) per day which is quite high considering averge transaction values are in the $10 – $100 range.

Anyway their success is maybe their undoing, because despite the phenominal growth, the service is even worse the electrictity availabiltity with the platform having an average uptime of 50% during normal working hours, after a 45 day downtime during November 1, 2011 – December 15, 2011 (which started as an upgrade then later turned into an outage).

From my software engineering background I am still baffled at why this continuously happens to one of the largest telco providers due to the established DevOps ( and practices: what are the possible solutions or approaches:

  1. High Transaction Volumes
    • Hardware – buy more hardware throw more power at the problem
    • Software – not scalable then run a cluster of boxes across the switches, load balance the sessions this problem is available even with HTTP
  2. Interface Operations – In database speak we usually state separate writes from reads. Separate balance checking (reads) from  withdrawals and deposits (writes) into separate distinct applications behind the interface. Use Queues, Gearman to ensure that the transactions are completed. Have the reads, balance checks run off slaves in the clusters …
  3. Notifications – SMS  Messages are good, for delivery but ensure they are sent and delivered. Queue the notifications so that they are always sent
  4. Provide options to execute transactions – provide a web interface for clients and agents. This opens up new revenue and agent opportunities since Internet cafe owners can also provide services from their interfaces. This is just an alternate way to access the service
  5. Be open to the public to lower the expectations – provide updates on service outages so that users do not just keep trying and only finding out from many failed trials. Failed transactions have been identified a known cause of application load spikes
  6. Reduce the number of available services and offload some services to other channels
  7. Use opensource software it has been proven to scale – or maybe some newer versions of your software applications
  8. New – Provide APIs so that developers can provide custom solutions to offload processing off your core system (switchboard)

These are just quick thoughts but they should be sufficient to start the discussion … not only rant and rave but also provide some concrete solutions

Doctrine2 – Day 1 – Commentary from the Trenches

The new year is here so it is time to get cracking on infrastructure for the year’s development projects. There are a number of things we have been putting off, but now that a number of our client hosts support PHP 5.3, it is the right time to move up the stack for our ORM from Doctrine 1.2 to Doctrine 2 (which feels more like Hibernate) than anything else.

Well step 1 was downloading it, I went straight to Github ( , got the Doctrine-ORM unpacked it dropped it into the library/Doctrine and was chugging away when I got errors. Seems like some classes were missing …

Tip #1: Download the ORM or DBAL packages from the main project website and you will get all required classes

Anyway classes in tow, I started cutting away at the Doctrine 1 classes to remove all that is not supported it is alot. Doctrine 1 was an Active Record implementation ( which meant that you had to extend a base class Doctrine_Record, while Doctrine 2 is a data mapper (, you write POPOs (Plain Old PHP Objects) for which you provide the mapping metadata so the database access is handled by a service layer.

We chose to use PHP Annotations over YAML and XML so that we do not introduce any dependencies and maintain the mapping metadata within the model classes.

Next step was integration with Zend Framework, so I followed the instructions at the Mayflower Blog (  with the following tweaks:

  1. Application.ini – instead of using doctrine.connectionParameters maintained the resources.db.params configuration which we use for storing sessions in the database using Zend_Session
  2. Bootstrap – load the resources.db settings and change configurations user to username and adapter to driver which Doctrine2 expects.
  3. Change production caching to ArrayCache since the shared hosts do not usually support APC, this will change on a project to project basis.

At least now the pages load now into the model configurations. We use two base classes BaseRecord was the one which extends Doctrine_Record in 1.2, and BaseEntity which provides commonly used properties id (autoincrement), datecreated, created by, lastupdate date and last updated by columns which are common to all tables.

First step was to add the mapping annotations, the column definitions are similar to Doctrine 1 with sensible defaults, which helps. However the associations were a mean piece of code, since the Doctrine 2 conventions are a total mismatch to normal database terminologies which I was used to. Opened a github issue too at (my first Github and Open Source issue)

Anyway what is interesting in the association mapping is that the mapping columns are defined in the annotation and private variables are not defined for the mapping columns (at least I do not see that in the examples) may come back to bite me later when I start saving the entities, but that is for another time

So anyway classes setup, now the next problem I ran into. In Doctrine 1, the Doctrine_Record provided two methods:

  1. Merge – which merges data into the object
  2. SynchronizeWithArray – same as merge, but had an added function, if you have relations defined and do not pass information in them, then the relations are removed, which was excellent for cases where you need to remove relations in a hurry

Where does this leave me now, action items are:

  1. Setup automated binding of an array of data to the object, and how to deal with relations …
  2. Update the application code to enable an object to load its details from the database … seems crude I will look at what others do
  3. Add Validators since Doctrine2 provides none. Now I need to poke into how I can use Zend_Validators without too much overhead and new changes for my team …

Anha one last one, I wanted to have a PDF of the documentation, since I wanted to use it on my PC faster than opening pages, but none was available. I pinged Jonathan Wage ( and he pointed me to Read the Docs ( which generates documentation for projects from Github etc, so you should check it out. Anyway I searched for Doctrine ORM documentation, downloaded a PDF and now I am in business.

All in a day’s work, and I hope to keep plodding and poking into it as promised.

My take on Day 1, the models feel light weight and more like POJOs, well I think PHP is becoming more and more enterprise ready as it supports core patterns of Enterprise Architecture a Martin Fowler religion that I subscribe to …

Till next time

ORM Anti-pattern – Right conclusion, wrong reasons – Rebuttal

This is a rebuttal for Right conclusion, wrong reasons  ( ) for an ORM being an anti-pattern.

The main purpose of an ORM (Object Relational Mapper) is to fix the object-relational mismatch, which is a mismatch between how an object is treated differently by the object oriented and relational paradigms which are fundamental constructs in programming and data storage/retrieval. The mismatch is well addressed by Scott Ambler of the Agile Manifesto (, Jeff Atwood of Coding Horror (

The traditional waterfall method handles this problem by trying to identify all entities and relationships before development begins. However the Rational Unified Process (RUP), Scrum and other agile development methods, however face a problem in that the data model changes, drastically during development, as more information becomes available and the client/implementation team learn more about the solution and business needs.

Hard-coding SQL statements becomes critical as these changes are not easily reflected as the database model evolves. ORMs like Hibernate ( and Doctine ( provide a means of mapping the POJOS (Plain Old Java Objects) and POPOs (Plain Old PHP Objects) into relational tables via XML mappings, and annotations (depending on your needs). As your model changes rapidly, all u have to keep changing is the database schema and object definitions.

The major strengths are also a big weakness, ORMs generate generic SQL which is not very optimized, so is slow when using traversing the object graphs, however a mix of ORM access (content pages, display pages where most attributes are displayed) and straight SQL (for lists and pagination) tends to reduce the impact of a the problem.

I have no experience in the new NoSQL data stores so I do not know what the implications are, but ease of unit testing, tool support etc, save time especially when the changes are rapid 🙂

MySQL Design and Administration Tools – Q1 2011

MySQL (http:/, the web database formely fully open source, then bought by Sun for $1bn, which was then swallowed by the all mighty database giant Oracle ( is finally growing up.

From MySQL 5.0 with the introduction of views and stored procedures (terrible performance) to 5.1 which stabilized the features to 5.5 (the latest GA) version which provides a true relational database experience. Oracle is positioning MySQL as the low-end database to compete against Microsoft SQL Server (, Postgre (, Firebird, Interbase etc, with an eventual upgrade to Oracle at the high end

As a database the tools for managing MySQL database have been “inferior” at best, or needing a commercial license of over $100. The most common tools are:

a) PhpMyAdmin ( – web based, opensource and free and one of the best available at the time

b) Adminer ( formerly PhpMinAdmin -web based, opensource and free -single file database management tool simpler and with less features than PhpMyAdmin

c) Navicat SQL ( – commercial GUI – probably the best MySQL GUI available

d) Webyog SQLyog ( commercial GUI – a similar feature set to Navicat, more powerful but not as userfriendly

e) MySQL Gui Tools ( – opensource GUI – administrator, Query Browser and workbench – free tools which were merged into MySQL Workbench 5.2 as a single tool

The latest MySQL workbench is making inroads into the territory for commercial tools, as it improves the quality of its adminstration, modelling and query tools.

Google Need for “Page” Speed – The Web Developer’s Arsenal

Google the king of search has made the speed of websites and page loading a key actor in ranking. This coupled with the need to use 3rd party frameworks for front end development, like JQuery, MooTools, Scriptalicious, Yahoo YUI, to support the multitiude of browsers has a larger burden web developers.

In true fashion however, the release of the Page Speed addons for both Firefox and Apache,, somewhat simplify this process.

We have developed a custom web application for a client, and it seemed that the pages were loading very slowly and performance was not acceptable. The approach that we are using to improve performance is three fold:

1. User Interface focused on browser improvements

– GZip all resources – using an Apache .htacess file

– Force caching of resources – expire all resources in the future, and require public cache (images, css) and private browser caching (javascript files)

– Minify CSS and Javascript files, and also combine each type into a single file to reduce the HTTP requests to load the page using Minify (

– Minify HTML contents when the page is loaded

2. Business Logic Enhancements

– Add an SQL lite cache to Doctrine resultsets and queries

– Reduce the number of components which are regenerated at the server back-end

3. Database Enhancements

– Use the smallest possible column types for each table – this is diffcult in some cases since the requirements are being identified as the application is being used

– Query performance – the tables have to be kept in 3rd normal form, therefore reporting performance will be affected. As the reports grow more complex, we will keep reviewing this

This is a continuous process so we keep measuring at each point, however each step leads us further into the depths of tuning and performance

The need for speed is now, the question is how far are u willing to go?

%d bloggers like this: