Backup Proposal

 

 

Documentation Revision History

Version

Date

Changes

0.5

05/30/00

Initial Release.  Needs a lot of work and review

 

 

 

 

 

 

 

 

 

 

1.    Introduction

Backup has become in increasing priority for Cobalt Products.  The following is designed to discuss the overall architecture of the Backup Solution with specific direction to the technical considerations for Carmel.

2.    Objectives

 

3.    Requirements

4.    Backup Areas

The issue of backup always brings up areas of discussion that are related to backup, but should be differentiated. A better overall solution is needed in helping the user get at data on their box when it is in an unstable state.  An overall solution also has to be provided for an ISP-type customer who wants to use their existing Backup Storage Vendor Provider to backup up multiple RaQs/Qubes.

 

a.      OS Restore CD Functionality

The OS Restore CD has the sole and complete responsibility of getting a RaQ into a sable and runable state if the customer is down.  The current OS Restore CD has limited functionality.  It contains the image of a product and can be used to restore a RaQ/Qube to a Factory Fresh state. Unfortunately, this is the only option to a customer if their machine is unresponsive. Since this is an integral tool for Tech support, it is desired to work to make the tool to be a little more robust to allow flexibility in getting customers up and running.  The following functionality would be desired.

·        Ability to restore RaQ/Qube from OS Restore CD to Factory state  (main function served now)

·        Ability to check disk Integrity of a RaQ/Qube from the OS restore CD

·        Ability to boot RaQ/Qube in somewhat of a single-user mode and ftp files in and out.

·        Ability to get diagnostic information that is being printed to console via an easy menu interface provided by an OS Restore CD.

·        Ability to run the OS Restore CD from a laptop

·        Ability to get RaQ to single-user mode and allow NFS mount of file systems to be look at by another system

It is important to note that the OS Restore CD should have no understanding of Backups and the format of the Backups.  The only use of the OS Restore CD is to get the customer in a stable state where they will be able to recovery information from their backups.

 

Who is going to be responsible for this area???????

b.     Backup/Recovery Core

This is the area that has the core level of functionality of what a backup consists of and how to recovery a backup.   This is knowledge that is specific to making a consistent backup of Cobalt Raqs and Qubes.  Even if a third party provider software is distributed on the Cobalt product, this is the software that will allow them to recover successfully any data that is specific to the Cobalt Products above file system backups.  This includes but is not limited to CCE database engine backup and recovery as well as postgress database backup and recovery.

 

·        UI: This is the display.  It will interface with a command line interface for doing backups.  The desire is to get away from doing backup in a cgi script.  The UI will just schedule backups and possibility display history information and display status.  This will fix the issue with browser timeouts as well as allow users to backup using cron

·        Keeps History:  The history of backups is important to

·        Uses dump or tar to perform Full and incremental backups.

c.      Media Interface

There are multiple Media vendors in the market today that do Backup Storage Management.  Cobalt does not want to get into this business.  It is necessary to provide limited capability to users so they can backup their data to disk and then recover without having to involve a huge Storage Solution provider to allow the small shops to backup Qube/RaQ data.

 There are two items that I would like to see accomplished.  First, the Cobalt Specific metadata information should be kept in a separate file and can be Backuped up separated from the file system backups.  This allows third party Backup vendors to backup Cobalt filesystems and then issue a command to backup (and later recovery) specific Cobalt configuration information. Second, Backups should be the same regardless of media.  This allows the user flexibility to extract a backup from tape, copy it to tape if needed and still be able to recover. Legato has been known to allow users to manually recover files from tape when they were unable to recovery automatically from their interface.

 

5.    Sausalito Backup Architecture

a.    Interface

The user will be able to do following

1.       Admin Full Backup

This will be the most commonly used type of backup.  It will backup the complete system with the exception

2.       Admin Incremental Backup

This will allow the user to backup files which have changed in the past ‘x’ days.

3.       User Backup

  This will allow a user to backup his/her own files and e-mail

 

b.    Scheduler

The scheduler is the interface between the UI and the backup/recovery modules.  It will kick off the backup and recovery.  The UI will not have a direct access to the backup files so that we can deal more with issues of  timeouts with browsers.

c.    Carmel Backup Algorithm

1.      Extract and Write out header information (see “Header Information” under “Backup File Contents”)

2.      Write out CCE Information ( NOTE: This will not be used for recovery in first version due to lack of robustness in the cce handlers.  Will backup the information and possibly use it for future versions.) 

If this is a user backup, only those objects within the users given namespace will be backed up.

1.      Issue ‘CLASSES’ command to CCE to get all classes

2.      Loop on each class ŕ Issue ‘Find’ command to CCE to get all object of each class

3.      Issue ‘Get’ command to CCE for each object in the database

4.      Get dependency information.:  (NOTE: This isn’t going to be there for first release.)  We need to build this into next release, so that user can recovery an object without recovery having to hard-code class dependencies.  This also probably will affect 3rd party apps using CCE which have dependencies on recovery.

5.      Get associated files:  (NOTE: This isn’t going to be there for first release) Issue command to handler to get files that this handler modifies in any way.  This will allow the backup routines to just backup files that are needed without backing up all of root.   These files should not be needed during recovery since the handler should make any relevant changes when it is created or modified during recovery.

3.      Backup CCE Database (NEEDED for First release)

Since we can use the logical backup done above because of issues

1.      Shutdown CCE

2.      Do tar/dump of /usr/Sausalito directory

3.      Restart CCE  

 

4.      Issue tar/dump of  /home directory (NOTE: Since we are not going to have full CCE recovery functionality, need to backup all files in root with the exception of log files.)

1.      This is a dump of the /home directory that will be referenced in the header information dumped.

2.      If this is an incremental, then it will determine which files need to be backed up.

3.      The file will span multiple files if need be.

 

6.    Recovery

a.    Interface

The user will be able to do following

1.       Admin Full Recovery

This will be the most commonly used type of backup.  It will backup the complete system with the exception

2.       Admin Selective restore

This will allow the user to selective restore files.  It will be done in a file tree manner so that we don’t have to worry about listing all files.

3.       User Recovery

This will allow a user to recover his/her own files and e-mail.  They can only restore those files which they owned because they backed them up.

b.    Carmel Recovery Algorithm

1.      Read header information (see “Header Information” under “Backup File Contents”)

1.      Verify user is same as who backed up

2.      Verify the state is same as backed up.  If not same, Warn user.

3.      Verify Dump files exist and are readable.

4.      Verify dependency information. If not same, Error.

2.      Recover CCE:  (This will change once handlers robust enough to handle

1.      Shutdown CCE

2.      Restore /usr/sausalito

3.      Restart CCE

3.      Issue tar/recover of  /home directory

1.      This will recover the /home directory that will be referenced in the header information dumped.

2.      If this is a selective restore, restore only the directories which user wants.

3.      The file may span multiple files.

 

 

7.    Backup File Contents

The current Cobalt back ups consist of file systems and files; there is no information about the backup or the system that it was backed up from.  To allow for more stringent sanity checks on recovery as well allow for an expandable solution, a new object-oriented backup format is proposed. The desire is to have a consistent format of the backup regardless of the product type or version.  The general format of the dump for a Qube should be the same as the RaQ.  How each portion of information is restored is dealt with on an individual level during restore.

Header Information

This information will be kept in XML format to allow for ease of viewing if needed by customer. 

·        Dump version: The protocol dump version, starting at 1.0.  Should not have to change too much.  If the format changes, this version will roll.  It has no correlation to any product information

·        Product Version number: The build type and version of the machine that this dump was taken of.  This can be gotten from /etc/build or from an Manufacturing number that may be put onto the box.

·        Machine Information: The machine information that this backup was taken from.

·        Dependency Version Object and number: Hook for specific product versioning information.  For example, Carmel will need to keep the CSCP (Cobalt System Configuration Protocol) version to check for CCE version capability.

·        Owner: the person who did the backup of this information.

·        Dump Type: The type of Dump being performed.  Expected values will be ‘admin’, ‘user’, ‘site-admin’.  This will also have a matrix for whether this is a Full or incremental dump. See Dump Types below.

01 = admin-Full

02 = admin-Incremental

05 = site-admin-Full

06=site-admin-Incremental

10 = user backup- Full

·        State Information: information to tell us which patches where on the machine at the time of backup to verify same state upon recovery. Currently that would be a list of everything that resides in the md5list directory.

·        Dependency information:  If this is an incremental dump, we need to store the name of the previous dump that it depends on.

·        Content information: Information about what command was used to invoke to produce this dump as well as where multiple dumps make up this dump.

·        Backup File location: The location of each backup file containing the dump of /home directory.

·        History information: A list of dumps done on this machine. See “Backup History” section.

Logical Content

The format of this area is going to be defined as we go along.  The only areas for the initial release will be the Object Database backup and the Logical User Information.

·        **Logical Service Information: Logical information for each service that is available on the box.  This is included in the backup of the object database in Carmel, but will have to be included in versions before Sausalito to backup and recovery information for service configuration.

·        Logical User Information: Logical meta information for a user.  In Carmel, this will be included in the CCE Object database dump.

·        **Logical Virtual Site Information: Logical meta information for a virtual site.  This allows for logical site migration.

·        Backup of Object Database:

The CCE Object database for Carmel needs to be backed up.

·        Backup of Postgress Database:

There is no consistent backup taken of the postgress database currently.  This leads to the inability to recover the database.

Physical Content

·        Physical File System Dump content: The physical dump of file system information that contains pretty much everything.  This can consists of multiple dumps that will be split out over the 2G limit.

**Third Party Content

Third Party  Dump content: This will provide a hook for third party vendors to backup data that is needed to restore that application.

 

** Designates will not be there for the first release with Carmel

8.    Backup History

It is important to start keeping history for a variety of reasons.  The most urgent is to allow tracking of incremental dumps.  The order and information contained in dumps is needed.  It would also be useful to display history information to the user as well as warning the user if they have not taken any backups of their system for a specified length of time. The backup history will be kept in /home/$user/.cbackup and the associated history file will be included in the backup.  The file is kept in /home, since if the user has multiple people doing backups and nobody cleans up, you don’t want it to affect the performance of the Cobalt machine.

 

 

 

 

 

 

 

 

 

 

 

 

 

 


State of Machine during Recovery

It is assumed that the user will be using the OS Restore CD to get the system

Back to a state where they can use the UI interface to recover their backup.

Issues of dealing with systems that are in an unstable state need to be addressed via the OS Restore group.

 

9.    TestCase Scenarios

The following are testcase scenarios that can be used by SQA to test Backup/Recovery.

(These testcases need to be signed off by Backup/Recovery team assembled from Tech Support and Product Marketing)

Minumium Backup/Recovery Functionality for Carmel GM.

Total Disaster recovery

1.      Do Full Admin Backup to Media from UI

2.      Do total OS Restore from CD

3.      Install any patches installed on Carmel that weren’t installed from OS Restore

4.      Recovery Full Admin Backup

Total Disaster recovery during heavy load

5.      Do Full Admin Backup to Media from UI while large load on system.  Particular where things are being added  or changed on the system. (i.e. add/delete users, maillists,  change settings of ssl etc.

6.      Do total OS Restore from CD

7.      Install any patches installed on Carmel that weren’t installed from OS Restore

8.      Recovery Full Admin Backup

 

Total Disaster recovery from multiple incremental backups

1.      Do Full Admin Backup to Media from UI

2.      Do Incremental Admin Backup to Media

3.      Do total OS Restore from CD

4.      Install any patches installed on Carmel that weren’t installed from OS Restore

5.      Recovery Full Admin Backups

Total Disaster recovery from backup spanning more than one 2 gig file

1.      Do Full Admin Backup to Media where backup is larger than 2G

2.      Do total OS Restore from CD

3.      Install any patches installed on Carmel that weren’t installed from OS Restore CD

4.      Recovery Full Admin Backup

Disaster recovery from scheduled backups for a full week

1.      Do Full Admin Backup to Media from UI

2.      Schedule incremental backups every day for a week.

3.      Do total OS Restore from CD

4.      Install any patches installed on Carmel that weren’t installed from OS Restore CD

5.      Recovery From last Backup.  Verify that system is restored starting from Full backup and with every subsequent backup

Disaster recovery of large system backup

1.      Do Full Admin Backup to Media from UI

2.      Do a backup of a system that will take multiple hours

3.      Do total OS Restore from CD

4.      Install any patches installed on Carmel that weren’t installed from OS Restore CD

5.      Recovery Backup.  Verify that no timeout from browser occurs.

User Backup and recovery of individual user complete /home information

1.      Do Full User Backup to Media

2.      Delete User home directory

3.      Restore User home directory

User Backup and Recovery of selective user information

1.      Do Full User Backup to Media

2.      Delete selective files from User home directory

3.      Restore selective files from User home directory

User recovery from Admin backup

6.      Do Full Admin Backup to Media

7.      Delete User

8.      Restore User

9.      Verify Home directory, e-mail list information restored as well as user configuration information.

 

Second Pass (Schedule as Patch after Carmel goes GM)

Restore of selective services which will translate down to

 

Recovery DNS configuration from Admin Backup

1.      Configure DNS of multiple machines

2.      Do Full Admin Backup to Media

3.      Change DNS / Remove DNS entries

4.      Restore DNS configuration service

5.      Verify DNS configuration has all records that existed through UI

 

Recovery Email configuration from Admin Backup

1.      Configure Email, add multiple users, multiple virtual sites

2.      Do Full Admin Backup to Media

3.      Delete Email configuration, in UI, modify sendmail files

4.      Restore email configuration

5.      Verify that sendmail files are restored along with all site and user email configuration.  Verify that mail received for all users and all sites

Recovery of a Virtual site from Admin Backup

1.      Configure virtual site.  Add users, website, and mail lists

2.      Do Full Admin Backup to Media

3.      Delete the virtual site

4.      Restore Virtual site configuration

5.      Verify webpage information as well as each users home directory and email configuration.