Lets say that at some point you decided to adhere to security best practices and set a password on your iPhone backups so that they are encrypted. A year or two later you have upgraded your iPhone to a new version and you want to transfer all of your data across to the new phone. You attempt to restore from your backup and, doh, you need to remember the password you set. You try every password you could have set but none of them work. You try to remove the "Encrypt iPhone Backup" option so that you can create a backup without a password, but again it asks you for the password to disable the password.
Here is a hack that will let you disable your encrypted backups without the password so that you can create an unencrypted version to use to transfer your data across to your new phone:
1. Jailbreak your device. The easiest way to do this is via www.jailbreakme.com, or if you have a more specific set of requirements then use Redsn0w or PwnageTool. This will install the hacked AppStore called Cydia.
2. Use Cydia to install OpenSSH and SQLite3
3. Connect your iPhone to a wireless network and ssh to it using Apple's default root password of "alpine"
4. We want to manipulate the keychains that basically store your saved encrypted passwords. Run "cd /var/Keychains" and you will find the keychains SQLite database called "keychain-2.db".
5. Connect to the keychain database by running the command "sqlite3 keychain-2.db"
6. Dump the contents of the table called "genp" by typing the following into the SQLite command prompt "select * from genp;"
7. Find the row containing the text "BackupPassword"
8. Get the RowID from the start of this line. Mine was "29"
9. Delete this row by typing the following into the SQLite command prompt "delete from genp where rowid = 29;" (remember to change 29 to be your own RowID)
10. Plug your iPhone into iTunes, where it will show that you still have a backup password enabled. Deselect the "Encrypt iPhone Backup" option.
11. This will ask you for your password, but don't fret! Type arbitrary text in as your password and hit ok.
At this point your password will be accepted, your backup encryption option will be disabled, and your phone will start backing up unencrypted.
Ref: Reverse Engineering Mac OS X