Cost Opimization for 1 scenario when backing to AWS for Commvault Backups

What is one way to optimize costs? – you can make a measure of both of how many copies of the data need to be copied per retention standard. With full deduplication, a copy measure of apprixmately 1.XX is a good measure of efficiency. In addition to using Copy Measure, you can always measure the cost in TB/Months. The lower it costs relative to other types of storage classes, the more efficient you can do your backups.

Ex. Our retention standard is 15 months. In order to minimize copies, you can create a new backup set in which types of backups occur at the following schedule: 1x Full every 15 month, in which it is retained for 30 months. The reason why it needs to be retained is that say after 15 months, if the full copy retention is deleted after 15 months, the differential chain will not work. As differential backups are cumulative backups of data, there can be a different retention standard for Differential of only 15 months. This way, It is optimized.

Ex. Our retention standard is 15 months. To be even more efficient and only store 1 copy of the full, you can create 1x full every 30 months and have a retention standard of 30 months.

As always, there are alway tradeoffs in using any approach.

Which tier is the Cheapest when it comes down to Cloud Storage Backups / Archives on AWS?

if you are okay with backups being restored within 12 hours, than you can consider the following. If the minimum of Commvault Backups is the Full’s Size, than that is the easiest way to calculate it over the long term, but there are additional tricks you can take in the cloud to spend even less.

Scenario: I would like to retain backups for 12 months in US-EAST-1. Which backup tier will cost the least? if the Cost from S3 Tiers from most expensive to least expensive per TB/month is $24, 12, 4, and 1 ( S3, S3 Inactive, Glacier, Glacier D.A). if we are using Deduplication/ Synthetic Fulls in this scenario, over 12 months the most ideal tier is S3 IA. As archive tiers cannot combine “Fulls” through Synthetic fulls, we simply to have to store more fulls times the number of months of retention.

Ex. S3 = $24 x 1 ( syn. full ), S3 IA ( syn. full ) = 12 x 1, S3 Glacier = 4 x 12, S3 Glacier D.A = 1 x 12. 12x from S3 IA is the same as S3 Glacier D.A, but it would make more sense to choose the tier that allows for faster restores. In this case, it would cost $12/TB a month in addition to running the Mediagent for about 7 days a month to make it a grand total of Cost = $20 ( M.A cost for stores less than 50 TB ) + $12 * the Amount of TB.

Scenario 2: Hey, its true that S3 IA can be the least expensive, but can you go even lower than that? – Yes, you can. If you use 1 backup set for local retention that has faster restores and simply just need a solution that allows for long term retention for compliance reasons, you could in theory just use the Commvault Combined S3 IA / Glacier Tier to get costs pretty low. You could get away with just having 1x Fulls in Glacier D.A. and get a cost factor of nearly 1.10x in the states and about 2.10x everywhere else.

Ex. i store 10 TB of Data. My costs could be as low as $10/month for storage and it will cost $20 to have the Commvault Media Agent only be on 7 days in that whole month for a total cost of $30/month to store the data. I will create 1 Full per a year and the rest of the months will utilize an Incremental backup.

Troubleshooting AWS AD Connector for AWS Chime

In troubleshooting AD connector, i learned the following:

The Primary requirements are:

-Open ports: 53, 88, 389 ( TCP/UDP)

-Service Account that is contained within that Domain ( mult-forest configuration is not supported on the AD Connector )

-Firewall rule allows for the system to connect to the DNS server

-Firewall rule allows for the system to connect to any Domain controller in that domain.

Other things to consider when migrating a domain from one account to another to make it work with AWS Chime:

-Only one domain of your forest and AD Connector directory service can be configured in AWS Chime. If you are using 1 e-mail domain worldwide, if you have 1 domain in each region, you would have to use 4 e-mail domain addresses as proxyaddresses for those users in order to authenticate these users worldwide.

-It mentions in the documentation that either the EmailAddress attribute or proxyaddress attribute can be used in that domain of the account. When it comes to migrating to another account, you cannot use the Proxyattribute approach on the users primary account as it has already been claimed as an active domain on the account you are trying to migrate away from. You must delete the domain from the old Chime account in order to make sure there is no conflicts in using the proxy approach.

Since the AD Connector can only be provisioned in AWS, when the service account queries for the IP address of a Domain Controller in that domain , it will than give the IP Address of any domain controller in that domain; even if there is a perfectly suitable Domain controller in the same subnet as the AD Connector. Since the Domain controller in which it queries from is Random, that makes creating the Firewall rules harder to constrain. What worked for us in this case is to temporarily open up the firewall rules so that the connection is not….random and to give it a chance to actually successfully connect. Once it successfully connects, we can think of limiting the firewall rules at that point. Logically we think sometimes it should route to the closest available system for its configuration, but sometimes the program just thinks: “I’m just gonna use any system as it is valid in your list!!!!”

AWS Networking: VPC Peering Vs. Resource Access Manager

Some of the interesting feature when using AWS is when you are trying to share resources either between different regions and different accounts. The following are some of the scenario’s in which you would use VPC Peering, Resource Access Manager (R.A.M) or both.

Assuming that you have a centralized resource that you would like to share:

  1. Is the resource and the destination to share in the same region in different accounts? – AWS RAM
  2. Is the resource and destination to share in a different region? – VPC Peering
  3. Is the resource and the destination to share in different accounts in the same region? – AWS RAM
  4. If R.A.M is used to share the subnet/network with a child account, if the resources are in different regions, you will generally need to: Share the resource subnet from the source region to the destination region using VPC Peering and than use ram to share the subnet with the child account in the same region.

Break Even point for using LTO6 Tapes compared to Cloud Archiving Systems using CommVault

Simplistic Assumptions: Your Physical Agents with Support costs $12.5K over 5 years and your HP MSL 4048 Tape Array costs $15.7K over 5 years. Each LTO Tapes costs $25. Iron Mountain Costs are assumed to be $1/per tape and includes the monthly Iron Mountain Service Costs

Simply: if you have 1 array, your break even point with a majority of Cloud Based systems in which price their Archive based tiers to about $1-4 / TB / Month, you would have to back up at least 500 – 600 TB comparably to make it worth it. Although, it does not include the labor costs related to having to physically load and unload a tape system. Although, if you have 3 offices your break even point would be 300 – 400 TB comparably at the low end and about ~ 1.5 PB at the end high end to make it cost competitive with cloud based systems.

Considerations when Backing up and Archiving in the Cloud

Here are some things to consider when backing up and Archiving Data in the Cloud

  1. Cost to Store the Data – generally you will use the following formula to calculate your Costs on either a monthly, yearly, or full-cycle Cost.
    • Cost per TB = [Cost per GB] X 100
    • Cost per Month = [Cost per TB] x 12
    • Full Lifecycle Cost = Cost TB x 60
  2. Cost to Restore the Data – Generally there is a Restore cost and a data transfer cost, depending on where it is restored.
    • ex. In AWS, it costs $0.09 / GB to restore when restored through the internet and $0.004 when restored through the VPN. If stored in the S3 Inactive Tier, it will cost $0.01 / GB to restore. In total, it will cost $0.05 / GB to restore through VPN and $0.10 to restore through the internet.
  3. Time to Restore data – Generally, the more expensive the data costs to restore, the less time it takes to restore said data. If you are comparing restore time from tape / using Iron Mountain for restore; Restore times within 1 day are usually acceptable. If you bemoan the thought of having to manually restore data for users and see that it grows on trees, you generally will want Active restore tiers and not Glacial / Archive Tiers.
    • AWS has the S3 – Inactive tier which is ideal for immediate restore for Archiving purposes. Generally data restore falls off after 6 years. It also costs 12x as much as S3 Glacier – Deep Archive.
    • AWS has the AWS S3 – Glacier – Deep Archive Tier which is optimized for storing data that is normally stored on Tapes. As it costs only $1 / TB / Month, use it as your eventual storage tier. Standard restore allows for restores within 12 hours while Bulk restore allows for restores within 48 hours.

4. Advanced – Storage Tiering to Optimize Long Term Costs – Generally, time the archiving tiering with the expected frequency of restore. For archiving purposes, you can perhaps store your data in the following ways. As Data restores normally fall off after 6 years, your archiving rules should A) archive data after 2 years, B) Store it in S3 Inactive for 3 years , C) store it on Glacier for another 2 years, and D) transition it to S3 Glacier – Deep Archive for the rest of the time the data is usable.

AWS Billing, Oh My!

Here are some of my thoughts on the use of Reserved Instances, Savings Plans, and such and such.

When should i use a Reserved Instance? – Once your on-demand Utilization of your computing instance on a yearly basis exceeds 33% of the year ( about 3 months for muggles ) or meets 50 – 70 percent of your base computing needs on a yearly basis.

Are Reserved Instances, in a multi-tenant environment, truly enforceable with no or partial upfront costs? – The answer is sadly…No. Although a user or tenant might agree to have a reserved instance for the year, in an AWS Organizations Scenario, if they simply delete their computing Instance, since there is no financial tagging mechanism to really follow through with that tagging till the end. The better and more enforceable mechanism is to a convertible, all upfront mechanism in which if do they decide to stop using, other users / agencies can benefit as it should be really a “use it, or lose it” scenario. Not a “Lets get married, and i might cheat later” kind of scenario.

When is it really practical to use Savings Plans? – As it doesn’t really define a particular minimum, it really only makes sense when you have a couple of machines in which they share the same computing instance family and it makes it a bit easier to shrimp or grow your instances. ex. instead of VCPU = 36 at 1 machine, perhaps 2 machines at 18 VCPU etc. etc.

Does it really make sense to use a Volume Gateway when using AWS Storage Gateway and Commvault? – if you are rolling in money and can’t be bothered as it is annoying, it might be worth it at that point. Otherwise, does it really make sense to pay $200 + dollars to retrieve 1 file from say that whole 3+ TB “Virtual Tape” Not really! Stick with using the File Share Gateway as S3 is more financially efficent in this case.

Is there a point of deploying an AWS Machine, when you don’t require External Access or require Infrastructure as Code? – Yes, spend more money than you have to be have a hipster and expensive machine in which you are renting.

Can you mange systems in AWS like you do with Traditional Infrastructure? – Why not, you didn’t care when it was hosted onsite, just spend 2-3x more to host it in AWS. Although, if you don’t like to move, its definitely worth it at that point!

What if the amount of running EC2 Instances exceeds my Reserved Instances? – It is magically on-demand and you are overpaying for it.

What is the best case scenario for sharing your Reserved instances in an AWS Organizations and Resource Access management Scenario? – If you implement an AWS Service Control Policy ( SCP ) to limit what types of EC2 Instance types can and cannot be deployed in an instance, a Reserved Instance or Saving plan can definitely help in this case.

What if i got 1 Reserved instances, which saves 70 percent, for say 3 on-demand instances? – Well, instead of paying 300 percent, you are now paying 230% for the same 3 machines.

Using a IR Probe on a BigTreeTech SKR 1.4 Board

Sometimes the best solutions…require out of the box thinking. Today, i was working on my 3D Printer and i was trying to adjust the IR Probe for use with the Z-Min Endstop probe. After running the M199 Comand ( get Endstop Status), i noticed that it would never trigger, even when the red LED status was on. The problem i was having was, “How come my Z-Endstop IR Probe will not trigger?” Directly from the Mini height Sensor board documentation, it mentions:

So…it says something about installing a pullup resistor. As you have to wait a week for shipping and you can’t purchase it from RadioShack as they went bankrupt back in the day, One solution that was posed was to change the pin-out to a port that i would likely never use. In this case i will simply change out the pins to a port in which does not have the 10K resistor installed…

and mainly change the Pins in my pins_BTT_SKR_V1_4.h file to use P1_25 instead of the original port of P1_27 in the documentation. That of course means i have to plug it into a different port…but it beats soldering!

Re-compiled the Firmware….and Presto!

As the Z-Endstop State was Triggered instead of being open, i have to just invert my Z-Endstop Logic State:

How do i find the Certificate Chain on a Third Party, Private GoDaddy SSL Certificate?

The problem i was having today was…What the hell is a Certificate Chain? As i just touched AWS Certificate Manager Today…i want to provide one solution to importing SSL Certificates into AWS Certificate Manager.

  1. Go to AWS Certificate Manager and click on Provision Certificates

2. Click on the blue heading on the top – Import a Certificate

3. Here is the question of the day – how do i fill it out?

Certificate Body – The contents of the .crt file, .pem file, or the contents between BEGIN CERTIFICATE and END CERTIFICATE. Note this may not the be the original BEGIN CERTIFICATE…END CERTIFICATE in your original request.

Certificate Private KEY – The Private Key corresponding to your Certificate. The vendor will not provide this to you; its something you saved already, right?

Certificate Chain – Since we got our Cert from GoDaddy, I looked up the keyword ‘GoDaddy Certificate Chain’ and Go this wonderful page: https://ssl-ccp.godaddy.com/repository?origin=CALLISTO

If you open the .crt file – you will notice it mentions Starfield Certificate Authority – G2. In this case, we will use the Starfield Certificate Bundle – G2 and Copy and Paste the Contents of that .crt file into our Certificate Chain Area