OK, back to this client. She has a successful company housed in an office in Irvine with about 10 user. Email and website are hosted elsewhere so the regular backup strategy is pretty straightforward and simple to set up and monitor. Put in a local NAS drive, set up scheduled nightly backups with Windows task scheduler and you’re done. Easy enough.

Now what do you do about off-site backups? This same client got broken into a few months ago and thieves took a laptop and a few other goodies, but thankfully they left the file server, the NAS drive, the switches, and other network components alone. So the urgency of doing offsite backups became woefully apparent. I have a couple of other clients using Mozy.com and I am happy with the results, although it does cost about a $1-2 per GB. So for this client the cost would be over $1,000 per year, which would be nice if we could mitigate. Furthermore, last time I had to restore file from Mozy, I had to call them and have them “prepare” a restore CD and FedEx it to us and that took about 3 days. I’m not too thrilled about that, although I like Mozy as a company and their software is pretty much fire and forget. Once you set it up right, it just works and it has nice features like bandwidth throttling and time of day scheduling and most importantly it runs as a Windows service not an executable, so you don’t have to stay logged on to the server or whatever machine is doing the backup.

So to make this short, the owner of this business lived near Irvine also and if you work in or around Central or South Orange County, you know that most residences as well as businesses have access to high-speed broadband (several times faster than a T1 in fact) on either DSL or cable/fiber. This client had 7.5MB download/1.5MB upload speeds at the office and roughly the same at her home. So what we ended up doing was set up two low-cost NetGear FVS114 VPN routers (which are less than $45 each on Ebay) one at the office and one at the owner’s residence and connected them them with a a site-to-site VPN tunnel. Then we purchased a Buffalo Terastation NAS drive (about $600), identical to the one already at the office and installed it at the owner’s residence.

Fig. 1: Site-to-Site VPN Network Diagram

For this client all the company accounting and user files reside on the NAS drive, which is configured with RAID 5 to protect against disk failures. There is no tape drive or tape library. Instead nightly and weekend backups are done from the NAS to the file server’s internal disk which is configured as a RAID 1 (mirroring). This configuration provides the added benefit that if the server were to crash, the users would still be able to access their files and get to the internet, so it would be more of a nuisance than a disaster.

Fig. 2: Netgear FVS114 Configuration

Fig 3: The two Terastations can see and communicate with each other

OK, back to the offsite backup. Once the site-to-site was set up, both NAS drives could see (ping, access, etc.) each other and the server could see them both. The Terastations have a nice backup and synchronization utility that allows you to back up one NAS to another automatically and this is what I enabled initially. BTW, Terastations run an embedded version of Linux inside so it has much more features than just a plain, dumb network disk drive, but you don’t have access to the OS, so the features and applications are not user tweakable. However it became evident that this was not going to work. First of all, the backup routine has no bandwidth throttling capability, so once you set it and kick start the service, it will hog the entire upload bandwidth and you’re going to have a lot of unhappy users. Secondly, there really isn’t any easy way to monitor the backup process to see if/when it failed and why. And the death nail to this approach was that, as far as I could tell, the backup was more of a synchronization – i.e. if files/folders were deleted from the office NAS, the deletions would propagate to the home NAS. Not good. What if a malicious user or a disgruntled employee got into your network and started deleting stuff and you didn’t find out about it until the next day? No, no no no. Bad. Very bad.

So to get around this I installed AllWaySync on the server and set up a “copy” operation from NAS1 to NAS2. This will create a mirror image on the home NAS. I set it to go off at night, every night, after the regular backup completes. I selected every night instead of Mon-Fri because many users log in via VPN and create files and make change from home or remote sites. I also made sure to select the “1-Way Left to Right” method and unchecked “Propagate Deletions” because you don’t want files either accidentally or intentionally deleted to also be deleted from the target NAS. This strategy has the added benefit that, let’s say ALL of the IT equipment in your office was lost, say a burglar broke in and took EVERYTHING but the kitchen sink. All that it would take to get back up and running is to bring the home NAS to the office, plug it into the network and voilà. The only tweaks you need to make is change the IP address and share drive mappings and you’re back in business. In minutes, not days. Of course this assumes your switch and cable modem/router were left alone, but if not, those are very easy to replace and can be easily found at local computer supply stores.

Fig. 4: AllWaySync Configuration

While we are going through this scenario, I can over-emphasize the importance of have an updated network topology diagram and inventory chart (stored off-site of course). For example in the above scenario, let’s say your cable modem was taken in the burglary. Even if you could replace it quickly, do you remember your public IP, logon account and password, DMZ or service or firewall settings? My guess would be that you wouldn’t and in a disaster recovery situation, that last thing you want to do is scramble to get information. So I highly recommend have a continually updated network diagram with all the IPs, accounts/passwords, screen caps, etc., and save it to a location where you can access it if all heck breaks loose. An of course password protect that file since you will have a lot of sensitive info.

Caveats

In configuring the site-to-site VPN, you may want to take some precaution in enabling the NetBIOS broadcasts since this will increase the volume of inter-site traffic, so if you have a relatively slow connection at either end, you may want to disable this in the VPN Policy screen. In the above example, bandwidth was not an issue and enabling NetBIOS allowed us to browse the remote network for drive mappings and other tasks.

Good luck with your disaster planning and offsite backups. Feel free to email me if you have questions about this setup.

One of the most common mistakes businesses, as well as individuals make is putting a backup system (e.g. tape or removable drives) in place and never testing to see if what is being backed up can be readily and reliably restored. Much like your insurance policy or the spare battery in your smoke alarm, if you don’t test the validity of your backup, by the time you find out you need it, it would be too late. In this article we will discuss some of the common techniques and best practices used by businesses today.

Backup Medium

• Tape – Although an aging technology which has changed little in the past three decades, backup to tape media is still the most common form used today. Tape has the advantage of long shelf life (2 – 4 yrs.), low cost and simple operation. Vendors such as IBM and Dell make tape solutions capable of backing up up to 1TB or more per tape and practically limitless size for tape libraries. Two of the major disadvantages of using tape are a) high cost of tape drives and tape libraries and b) Slow backup and restore times.

• Disk – Backing up to a disk or more likely a disk array, has become very popular in recent years, primarily because of the declining price of disk drives and the speed of backups and restores.
• Removable Media – Many small business use removable disk drives such as iOmega to back up their critical files. the major shortcoming of the method is the limited space typically offered by such media and the fact that it is a manual process and requires discipline to do it consistently.
• On-Line – Many business and individuals are turning to on-line backup services such as Mozy and HP. The greatest advantage of this method is physical separation (discussed below). However to be done properly and reliably, at a minimum you need to have T-1 connection speeds and the backup needs to carefully configured to not overwhelm the bandwidth (which it shares will all the users). Most on-line backup service providers charge on a per-Gigabyte basis. In other words, the more data you have to back up the more you pay. In recent years Amazon and iDrive have come up with relatively low cost or free solutions, however these services are primarily targeted to individual users, not businesses, as they are either limited in the size of backup allowed, or are not compatible to be run on servers where business data usually resides.

Rotation Strategies

• Grand Father, Father, Son – This is the most common strategy used to rotate tapes or “hives”. In this strategy you label tapes for daily, weekly and monthly rotation and as you go though one set of daily tapes (sons), you graduate the last tape to the weekly (father) status and so forth. Some users make a single full backup, e.g. on Mondays and incremental or differential backup on subsequent days. That is not a wise strategy because if something were to have to your full backup tape, you would have no way of recovering most of your files. Tip: Use disk-based backups for daily runs and use tape for end of week.
• Daily Rotation – In this method five or more tapes are used, one for each day of the week and rotated each week. This not a cost-effective or efficient method
• Continuous Backup – First introduced by Veritas Backup Exec, continuous backup is exactly what is sounds like. In other words, the software monitors files and other data on the servers and whenever it sees a change, it backs it up and creates “snapshot” of the file at that point in time. So let’s say you are working on a spreadsheet and you save it ten times during the day as you modify it. With continuous backup, you could conceivable go back to each of the ten versions and recover that specific version when and if you need it.

Physical Security

Many businesses host or house their critical servers and backup system at a data center. This is a prudent strategy as most data centers have a array of redundancies such as uninterruptible power supplies (UPS), structural and access security and redundant internet connections. However just because your servers are in a secure location doesn’t mean you can neglect the other aspects of a backup strategy. After all if you had an un-authorized access to your servers and some data was lost or corrupted, the physical security of the data center will be of little help. Furthermore, living in an earthquake prone state here in California, it is not inconceivable that even a mild earthquake might render your server and your backups inaccessible, at least temporarily. So having a robust rotation strategy and “pulling” tapes is still necessary even if your server are safe. The idea of pulling tapes is to create a physical separation between where your data resides and there the backup tapes or disks are stored.

Another factor to consider is this. Let’s say you have been pulling and saving your tapes religiously and all of a sudden your servers were lost in a fire. Even if you have the backup tapes, do you have the tape drives and the software to recover the data on those tapes? The answer is simple. You need to have a method to quickly recover your data. This could be as simple as having a spare tape drive at home or an online “life-line”. Keep in mind that most online backup services do not guarantee how quickly you can access and restore your files, especially if you have a large volume of files or databases. For example in the case of Mozy.com, your request to burn a 20GB database to or a DVD or tape and send it to you will take several days.

Business Objective

In almost any type of business today, the goal of continuity of business relies on the availability and quick recovery in case of a disaster, of the critical files and databases needed to run the business. This includes your MS Office files, your emails, your SQL databases that might store everything from your customer data to you accounting system. Backing up of data is, or should be, only part of the overall disaster recovery plan. Most businesses who are genuinely committed to this principle have at least two backup methodologies. For example, the primary backup may be done to a disk array on a continuous or nightly basis and second on-line backup provides a “life-line” in case something were to happen to the servers and data storage arrays. To further safeguard the process, a tape (or whatever medium is used) is pulled and stored off-site at least once a week. The idea is to never have less backup sources than you can afford to lose. If you can afford to lose up to one week of files and updates without it affecting your business, then pulling one tape a week will probably be adequate. If you have a 24×7 business and your customers rely on your web and database server to be up all the time, then you probably need to implement a more robust disaster recovery plan.