Update: CDBA has been updated into a product, DeltaV Backup and Recovery.
Original post: Twice here at Emerson Process Experts, I’ve featured the work of Shenling Yang. The first was in her role as member of the DeltaV technology team and the second as an integration specialist in the Life Sciences industry center. Shenling shared with me a presentation she is developing with a biotech manufacturer for the Emerson Exchange on backing up and recovering critical process data. This is a huge requirement for regulated industries like pharmaceutical and biotech manufacturers.
The scenario they will present is the dreaded 3 am phone call from the plant with the news that production has stopped, people standing around and it’s up to you to do something. Choice one is to go to the plant, to rebuild the automation system configuration, to revalidate the process, to lose a bioreactor batch that may have been running for up to 100 days, and then to hopefully resume production within a few weeks. To give you a sense of the value of saving a batch, it’s important to note that the medications being produced in these bioreactors save lives and any loss of a batch means a delay for a patient who needs this medication.
Choice two is to load the data backed up from the critical data backup application, have the operator restart the plant, verify normal operations and save the running batch in the bioreactor. Obviously, choice two was the way to go. It is vital to protect and recover control system data because human error or system failure can wipe out years of work, experience, plant operations information and process records.
The U.S. Food and Drug Administration (FDA) regulates and requires fully validated backup and restoration solutions for critical data. With the FDA’s 21CFR Part 11 electronic record provisions include the accurate and ready retrieval of control system information through the record retention period. The FDA’s 21CFR Part 210 & 211 good manufacturing practices (GMP) require this retention period be at least one year after the expiration date of the batch.
The goal established by this biotech manufacturer was to be able to recover and be back up and running 100% with minimal loss in three hours or less. They were looking for something with minimal customization that would automatically back up the configuration and version control databases without any operator intervention. The solution was to use the Critical Data Backup application (CDBA) developed by the data management services team to meet the 21CFR Part 11 compliance for backup, recovery and preservation of electronic records. It’s a part of the overall disaster recovery plan, which includes files, spare on-site server hardware, physical separation of equipment and networks, and always-available support personnel on-site and at their Emerson local business partner location.
The backup system includes a server, tape carousel and gigabit network to link multiple DeltaV systems and transfer large files quickly, safely and efficiently. This application help formalize the backup process which was not as diligent or documented as it should have been.
Like anyone who administers a server or even backs up the family PCs can attest, you don’t know how good it will work until you have to use it. The day came when this site lost two hard drives in a RAID array. They were able to put a new server in the rack, transfer data using CDBA, do a warm batch restart and be back up in an hour with no loss to the batch.
This sounds like a great presentation to catch if you need a way to formalize your system backup and recovery effort.