Posts Tagged ‘data mapper’

15 Apr

An Opinionated Approach to OpenMRS Concept Management

Posted by ssmusoke in eHealth, openmrs, opensource, software delivery, software development lifecycle, tips and tricks. Tagged: , , .

One of the key strengths of the OpenMRS platform is the concept dictionary, which allows for the mapping of real world health care data needs into concepts that provide the questions and answers.

The concept dictionary provides ability to map real life concepts to specializations across health care domains such as: SNOMED CT (clinical healthcare terminology), LOINC (laboratory observations), ICD-10/11 (disease classifications), RxNORM (normalized names for clinical drugs), CVX (vaccination codes) in addition to leaders in certain medical fields such as Partners In Health and AMPATH (HIV Care and treatment).

However with great power comes great responsibility, the concept dictionary coding can easily get out of hand, with duplicate concepts leading to inability to extract data for reporting and improving the efficiency clinical care that are key goals of health informatics activities.

This guide is based on my personal experience from working across multiple diverse implementations and the fact that Open Concept Lab (OCL) is not yet in widespread production usage which would alleviate most of the pains, which includes:

  1. Supporting the upgrade, evolution and rollout of UgandaEMR in Uganda from 350 sites in 2016 to over 1000 sites (December 2019) including implementation of 2 major revisions of national Health Management Information Systems tools
  2. Migration of 2 custom OpenMRS implementations in Uganda to align to and build on top of UgandaEMR
  3. Namibia PTracker PMTCT program
  4. Enhancements of the Reference Application

The key principles to this approach are as follows:

  1. CIEL dictionary is the official source of concepts, and first place to check for concepts
  2. Custom concepts must be setup in such a way that updates to CIEL or other custom modules used do not overwrite the customizations

Concept Server Setup

The setup involves using the following OpenMRS instances:

  1. CIEL dictionary server running the latest reference application version – contains the latest version of the CIEL dictionary and helps for data export when needed. The complete concept dictionary however is not recommended to be loaded for an implementation due to slowdowns in concept lookup, as sample concept numbers are:
    • Reference Application 2.9.0 – 446 concepts
    • UgandaEMR – 5,500 concepts
    • CIEL dictionary – 54,000 concepts
  2. Custom Concepts Server – this one contains the custom concepts and CIEL concepts that exist, is the single source of truth for the project concepts. This needs to run similar modules to what the implementation is running. I would recommend the following additional steps:
    • Set the autoincrement value of the concept table to 5,000,000. This ensure that the concepts created will never be overwritten by CIEL concepts, the last numeric ID as of April 2020 is 165900 due to retired concepts etc
    • Set the auto increment values of the rest of the concept* tables to 10,000,000 (there are usually more values than so these tables grow)
    • Create a custom mapping for your project or implementation, and use that to reference the custom and CIEL concepts that you use in your forms and reports. This adds a layer of redirection and consistency for access
  3. Implementation Development, Staging and Demo Servers – as needed

Moving Concepts Across Servers

There are multiple options for moving concepts from the CIEL to the Custom Concepts Server and finally to the implementation servers

Approach Description Notes
Use of Metadata sharing and metadata deploy modules to build of metadata packages (zip files)
  • Manually downloaded from the servers and uploaded where needed – faces the challenges of manual processes
  • Provide a dedicated url from source servers that clients can subscribe to get updates as the deploy packages are updated following the pub-sub model
  • The creation of the metadata packages is manual
  • The metadata packages are zip files which are difficult to debug
Download the concept data into CSV or DB Unit compliant XML The concept files are then loaded into the implementation using either Initializer or Data Exchange modules
  • This requires developers to extract the concept data from the different tables, without making mistakes.
  • This is still a manual and error prone process
Open Concept Lab (OCL) Allows creation of custom dictionaries and extraction of data through an online web interface This tool is not yet production ready but would provide the automation that solves all issues

Data Export Tools

The following additional tools can be leveraged for this purpose

  1. Dbunit XML data export plugin for Data Grip – https://ssmusoke.com/2017/04/17/techtip-dbunit-export-from-jetbrains-datagrip/

 

 

5 Feb

Doctrine2 – Day 1 – Commentary from the Trenches

Posted by ssmusoke in database, doctrine2, opensource, php, software development lifecycle, web development, xml, Zend Framework. Tagged: , , , , , , , .

The new year is here so it is time to get cracking on infrastructure for the year’s development projects. There are a number of things we have been putting off, but now that a number of our client hosts support PHP 5.3, it is the right time to move up the stack for our ORM from Doctrine 1.2 to Doctrine 2 (which feels more like Hibernate) than anything else.

Well step 1 was downloading it, I went straight to Github (https://github.com/doctrine) , got the Doctrine-ORM unpacked it dropped it into the library/Doctrine and was chugging away when I got errors. Seems like some classes were missing …

Tip #1: Download the ORM or DBAL packages from the main project website and you will get all required classes

Anyway classes in tow, I started cutting away at the Doctrine 1 classes to remove all that is not supported it is alot. Doctrine 1 was an Active Record implementation (http://en.wikipedia.org/wiki/Active_record) which meant that you had to extend a base class Doctrine_Record, while Doctrine 2 is a data mapper (http://martinfowler.com/eaaCatalog/dataMapper.html), you write POPOs (Plain Old PHP Objects) for which you provide the mapping metadata so the database access is handled by a service layer.

We chose to use PHP Annotations over YAML and XML so that we do not introduce any dependencies and maintain the mapping metadata within the model classes.

Next step was integration with Zend Framework, so I followed the instructions at the Mayflower Blog (http://blog.mayflower.de/archives/799-06.12.-Doctrine-2-Zend-Framework-Integration.html)  with the following tweaks:

  1. Application.ini – instead of using doctrine.connectionParameters maintained the resources.db.params configuration which we use for storing sessions in the database using Zend_Session
  2. Bootstrap – load the resources.db settings and change configurations user to username and adapter to driver which Doctrine2 expects.
  3. Change production caching to ArrayCache since the shared hosts do not usually support APC, this will change on a project to project basis.

At least now the pages load now into the model configurations. We use two base classes BaseRecord was the one which extends Doctrine_Record in 1.2, and BaseEntity which provides commonly used properties id (autoincrement), datecreated, created by, lastupdate date and last updated by columns which are common to all tables.

First step was to add the mapping annotations, the column definitions are similar to Doctrine 1 with sensible defaults, which helps. However the associations were a mean piece of code, since the Doctrine 2 conventions are a total mismatch to normal database terminologies which I was used to. Opened a github issue too at https://github.com/doctrine/orm-documentation/issues/80 (my first Github and Open Source issue)

Anyway what is interesting in the association mapping is that the mapping columns are defined in the annotation and private variables are not defined for the mapping columns (at least I do not see that in the examples) may come back to bite me later when I start saving the entities, but that is for another time

So anyway classes setup, now the next problem I ran into. In Doctrine 1, the Doctrine_Record provided two methods:

  1. Merge – which merges data into the object
  2. SynchronizeWithArray – same as merge, but had an added function, if you have relations defined and do not pass information in them, then the relations are removed, which was excellent for cases where you need to remove relations in a hurry

Where does this leave me now, action items are:

  1. Setup automated binding of an array of data to the object, and how to deal with relations …
  2. Update the application code to enable an object to load its details from the database … seems crude I will look at what others do
  3. Add Validators since Doctrine2 provides none. Now I need to poke into how I can use Zend_Validators without too much overhead and new changes for my team …

Anha one last one, I wanted to have a PDF of the documentation, since I wanted to use it on my PC faster than opening pages, but none was available. I pinged Jonathan Wage (http://www.jwage.com/) and he pointed me to Read the Docs ( http://readthedocs.org/) which generates documentation for projects from Github etc, so you should check it out. Anyway I searched for Doctrine ORM documentation, downloaded a PDF and now I am in business.

All in a day’s work, and I hope to keep plodding and poking into it as promised.

My take on Day 1, the models feel light weight and more like POJOs, well I think PHP is becoming more and more enterprise ready as it supports core patterns of Enterprise Architecture a Martin Fowler religion that I subscribe to …

Till next time