Cost Opimization for 1 scenario when backing to AWS for Commvault Backups

What is one way to optimize costs? – you can make a measure of both of how many copies of the data need to be copied per retention standard. With full deduplication, a copy measure of apprixmately 1.XX is a good measure of efficiency. In addition to using Copy Measure, you can always measure the cost in TB/Months. The lower it costs relative to other types of storage classes, the more efficient you can do your backups.

Ex. Our retention standard is 15 months. In order to minimize copies, you can create a new backup set in which types of backups occur at the following schedule: 1x Full every 15 month, in which it is retained for 30 months. The reason why it needs to be retained is that say after 15 months, if the full copy retention is deleted after 15 months, the differential chain will not work. As differential backups are cumulative backups of data, there can be a different retention standard for Differential of only 15 months. This way, It is optimized.

Ex. Our retention standard is 15 months. To be even more efficient and only store 1 copy of the full, you can create 1x full every 30 months and have a retention standard of 30 months.

As always, there are alway tradeoffs in using any approach.

Which tier is the Cheapest when it comes down to Cloud Storage Backups / Archives on AWS?

if you are okay with backups being restored within 12 hours, than you can consider the following. If the minimum of Commvault Backups is the Full’s Size, than that is the easiest way to calculate it over the long term, but there are additional tricks you can take in the cloud to spend even less.

Scenario: I would like to retain backups for 12 months in US-EAST-1. Which backup tier will cost the least? if the Cost from S3 Tiers from most expensive to least expensive per TB/month is $24, 12, 4, and 1 ( S3, S3 Inactive, Glacier, Glacier D.A). if we are using Deduplication/ Synthetic Fulls in this scenario, over 12 months the most ideal tier is S3 IA. As archive tiers cannot combine “Fulls” through Synthetic fulls, we simply to have to store more fulls times the number of months of retention.

Ex. S3 = $24 x 1 ( syn. full ), S3 IA ( syn. full ) = 12 x 1, S3 Glacier = 4 x 12, S3 Glacier D.A = 1 x 12. 12x from S3 IA is the same as S3 Glacier D.A, but it would make more sense to choose the tier that allows for faster restores. In this case, it would cost $12/TB a month in addition to running the Mediagent for about 7 days a month to make it a grand total of Cost = $20 ( M.A cost for stores less than 50 TB ) + $12 * the Amount of TB.

Scenario 2: Hey, its true that S3 IA can be the least expensive, but can you go even lower than that? – Yes, you can. If you use 1 backup set for local retention that has faster restores and simply just need a solution that allows for long term retention for compliance reasons, you could in theory just use the Commvault Combined S3 IA / Glacier Tier to get costs pretty low. You could get away with just having 1x Fulls in Glacier D.A. and get a cost factor of nearly 1.10x in the states and about 2.10x everywhere else.

Ex. i store 10 TB of Data. My costs could be as low as $10/month for storage and it will cost $20 to have the Commvault Media Agent only be on 7 days in that whole month for a total cost of $30/month to store the data. I will create 1 Full per a year and the rest of the months will utilize an Incremental backup.

Troubleshooting AWS AD Connector for AWS Chime

In troubleshooting AD connector, i learned the following:

The Primary requirements are:

-Open ports: 53, 88, 389 ( TCP/UDP)

-Service Account that is contained within that Domain ( mult-forest configuration is not supported on the AD Connector )

-Firewall rule allows for the system to connect to the DNS server

-Firewall rule allows for the system to connect to any Domain controller in that domain.

Other things to consider when migrating a domain from one account to another to make it work with AWS Chime:

-Only one domain of your forest and AD Connector directory service can be configured in AWS Chime. If you are using 1 e-mail domain worldwide, if you have 1 domain in each region, you would have to use 4 e-mail domain addresses as proxyaddresses for those users in order to authenticate these users worldwide.

-It mentions in the documentation that either the EmailAddress attribute or proxyaddress attribute can be used in that domain of the account. When it comes to migrating to another account, you cannot use the Proxyattribute approach on the users primary account as it has already been claimed as an active domain on the account you are trying to migrate away from. You must delete the domain from the old Chime account in order to make sure there is no conflicts in using the proxy approach.

Since the AD Connector can only be provisioned in AWS, when the service account queries for the IP address of a Domain Controller in that domain , it will than give the IP Address of any domain controller in that domain; even if there is a perfectly suitable Domain controller in the same subnet as the AD Connector. Since the Domain controller in which it queries from is Random, that makes creating the Firewall rules harder to constrain. What worked for us in this case is to temporarily open up the firewall rules so that the connection is not….random and to give it a chance to actually successfully connect. Once it successfully connects, we can think of limiting the firewall rules at that point. Logically we think sometimes it should route to the closest available system for its configuration, but sometimes the program just thinks: “I’m just gonna use any system as it is valid in your list!!!!”

AWS Networking: VPC Peering Vs. Resource Access Manager

Some of the interesting feature when using AWS is when you are trying to share resources either between different regions and different accounts. The following are some of the scenario’s in which you would use VPC Peering, Resource Access Manager (R.A.M) or both.

Assuming that you have a centralized resource that you would like to share:

  1. Is the resource and the destination to share in the same region in different accounts? – AWS RAM
  2. Is the resource and destination to share in a different region? – VPC Peering
  3. Is the resource and the destination to share in different accounts in the same region? – AWS RAM
  4. If R.A.M is used to share the subnet/network with a child account, if the resources are in different regions, you will generally need to: Share the resource subnet from the source region to the destination region using VPC Peering and than use ram to share the subnet with the child account in the same region.

Break Even point for using LTO6 Tapes compared to Cloud Archiving Systems using CommVault

Simplistic Assumptions: Your Physical Agents with Support costs $12.5K over 5 years and your HP MSL 4048 Tape Array costs $15.7K over 5 years. Each LTO Tapes costs $25. Iron Mountain Costs are assumed to be $1/per tape and includes the monthly Iron Mountain Service Costs

Simply: if you have 1 array, your break even point with a majority of Cloud Based systems in which price their Archive based tiers to about $1-4 / TB / Month, you would have to back up at least 500 – 600 TB comparably to make it worth it. Although, it does not include the labor costs related to having to physically load and unload a tape system. Although, if you have 3 offices your break even point would be 300 – 400 TB comparably at the low end and about ~ 1.5 PB at the end high end to make it cost competitive with cloud based systems.

Considerations when Backing up and Archiving in the Cloud

Here are some things to consider when backing up and Archiving Data in the Cloud

  1. Cost to Store the Data – generally you will use the following formula to calculate your Costs on either a monthly, yearly, or full-cycle Cost.
    • Cost per TB = [Cost per GB] X 100
    • Cost per Month = [Cost per TB] x 12
    • Full Lifecycle Cost = Cost TB x 60
  2. Cost to Restore the Data – Generally there is a Restore cost and a data transfer cost, depending on where it is restored.
    • ex. In AWS, it costs $0.09 / GB to restore when restored through the internet and $0.004 when restored through the VPN. If stored in the S3 Inactive Tier, it will cost $0.01 / GB to restore. In total, it will cost $0.05 / GB to restore through VPN and $0.10 to restore through the internet.
  3. Time to Restore data – Generally, the more expensive the data costs to restore, the less time it takes to restore said data. If you are comparing restore time from tape / using Iron Mountain for restore; Restore times within 1 day are usually acceptable. If you bemoan the thought of having to manually restore data for users and see that it grows on trees, you generally will want Active restore tiers and not Glacial / Archive Tiers.
    • AWS has the S3 – Inactive tier which is ideal for immediate restore for Archiving purposes. Generally data restore falls off after 6 years. It also costs 12x as much as S3 Glacier – Deep Archive.
    • AWS has the AWS S3 – Glacier – Deep Archive Tier which is optimized for storing data that is normally stored on Tapes. As it costs only $1 / TB / Month, use it as your eventual storage tier. Standard restore allows for restores within 12 hours while Bulk restore allows for restores within 48 hours.

4. Advanced – Storage Tiering to Optimize Long Term Costs – Generally, time the archiving tiering with the expected frequency of restore. For archiving purposes, you can perhaps store your data in the following ways. As Data restores normally fall off after 6 years, your archiving rules should A) archive data after 2 years, B) Store it in S3 Inactive for 3 years , C) store it on Glacier for another 2 years, and D) transition it to S3 Glacier – Deep Archive for the rest of the time the data is usable.

AWS Billing, Oh My!

Here are some of my thoughts on the use of Reserved Instances, Savings Plans, and such and such.

When should i use a Reserved Instance? – Once your on-demand Utilization of your computing instance on a yearly basis exceeds 33% of the year ( about 3 months for muggles ) or meets 50 – 70 percent of your base computing needs on a yearly basis.

Are Reserved Instances, in a multi-tenant environment, truly enforceable with no or partial upfront costs? – The answer is sadly…No. Although a user or tenant might agree to have a reserved instance for the year, in an AWS Organizations Scenario, if they simply delete their computing Instance, since there is no financial tagging mechanism to really follow through with that tagging till the end. The better and more enforceable mechanism is to a convertible, all upfront mechanism in which if do they decide to stop using, other users / agencies can benefit as it should be really a “use it, or lose it” scenario. Not a “Lets get married, and i might cheat later” kind of scenario.

When is it really practical to use Savings Plans? – As it doesn’t really define a particular minimum, it really only makes sense when you have a couple of machines in which they share the same computing instance family and it makes it a bit easier to shrimp or grow your instances. ex. instead of VCPU = 36 at 1 machine, perhaps 2 machines at 18 VCPU etc. etc.

Does it really make sense to use a Volume Gateway when using AWS Storage Gateway and Commvault? – if you are rolling in money and can’t be bothered as it is annoying, it might be worth it at that point. Otherwise, does it really make sense to pay $200 + dollars to retrieve 1 file from say that whole 3+ TB “Virtual Tape” Not really! Stick with using the File Share Gateway as S3 is more financially efficent in this case.

Is there a point of deploying an AWS Machine, when you don’t require External Access or require Infrastructure as Code? – Yes, spend more money than you have to be have a hipster and expensive machine in which you are renting.

Can you mange systems in AWS like you do with Traditional Infrastructure? – Why not, you didn’t care when it was hosted onsite, just spend 2-3x more to host it in AWS. Although, if you don’t like to move, its definitely worth it at that point!

What if the amount of running EC2 Instances exceeds my Reserved Instances? – It is magically on-demand and you are overpaying for it.

What is the best case scenario for sharing your Reserved instances in an AWS Organizations and Resource Access management Scenario? – If you implement an AWS Service Control Policy ( SCP ) to limit what types of EC2 Instance types can and cannot be deployed in an instance, a Reserved Instance or Saving plan can definitely help in this case.

What if i got 1 Reserved instances, which saves 70 percent, for say 3 on-demand instances? – Well, instead of paying 300 percent, you are now paying 230% for the same 3 machines.

Using a IR Probe on a BigTreeTech SKR 1.4 Board

Sometimes the best solutions…require out of the box thinking. Today, i was working on my 3D Printer and i was trying to adjust the IR Probe for use with the Z-Min Endstop probe. After running the M199 Comand ( get Endstop Status), i noticed that it would never trigger, even when the red LED status was on. The problem i was having was, “How come my Z-Endstop IR Probe will not trigger?” Directly from the Mini height Sensor board documentation, it mentions:

So…it says something about installing a pullup resistor. As you have to wait a week for shipping and you can’t purchase it from RadioShack as they went bankrupt back in the day, One solution that was posed was to change the pin-out to a port that i would likely never use. In this case i will simply change out the pins to a port in which does not have the 10K resistor installed…

and mainly change the Pins in my pins_BTT_SKR_V1_4.h file to use P1_25 instead of the original port of P1_27 in the documentation. That of course means i have to plug it into a different port…but it beats soldering!

Re-compiled the Firmware….and Presto!

As the Z-Endstop State was Triggered instead of being open, i have to just invert my Z-Endstop Logic State:

How do i find the Certificate Chain on a Third Party, Private GoDaddy SSL Certificate?

The problem i was having today was…What the hell is a Certificate Chain? As i just touched AWS Certificate Manager Today…i want to provide one solution to importing SSL Certificates into AWS Certificate Manager.

  1. Go to AWS Certificate Manager and click on Provision Certificates

2. Click on the blue heading on the top – Import a Certificate

3. Here is the question of the day – how do i fill it out?

Certificate Body – The contents of the .crt file, .pem file, or the contents between BEGIN CERTIFICATE and END CERTIFICATE. Note this may not the be the original BEGIN CERTIFICATE…END CERTIFICATE in your original request.

Certificate Private KEY – The Private Key corresponding to your Certificate. The vendor will not provide this to you; its something you saved already, right?

Certificate Chain – Since we got our Cert from GoDaddy, I looked up the keyword ‘GoDaddy Certificate Chain’ and Go this wonderful page: https://ssl-ccp.godaddy.com/repository?origin=CALLISTO

If you open the .crt file – you will notice it mentions Starfield Certificate Authority – G2. In this case, we will use the Starfield Certificate Bundle – G2 and Copy and Paste the Contents of that .crt file into our Certificate Chain Area

Amazon AWS Governance Reminders and Billing

Tips and Tricks to help with Administering an AWS Instance and helping getting infrastructure paid for in a clear way.

FAQS / Tips

  1. Account Separation and Billing – the most important thing to do in AWS is separate group’s by whom in the organization is willing to pay for said infrastructure. If you have a developer group, they get one account. If the group is in another continent, they get another account. etc. etc. If you don’t know how to chargeback costs, at least it can be paid back on an account level.

2. Use Tagging to allocate Billing by Tags – Whether this be done by project or organization code, anything that can cost a significant amount of money should use a tag. The use of AWS Config can we used to ensure all relevant resources are tagged properly.

3. Shutdown all Compute Resources after 2 hours if they do not have the correct tagging – Especially for unapproved resources, create a no tolerance policy on tagging. Shut down all resources that do not follow this tagging mandate.

4. Limit Accounts, especially is using AWS Organizations, by minimal service and Region Policies – Implement guardrails that allow users to not access IAM, limit to known services, limit to known deployment regions, and limit to known certain subnets as part of the VPC.

5. Implement guardrails from prevent users from disabling cloudtrails and cloudwatch at the AWS Organization level – As cloudwatch monitors resources and Cloudtrails monitors API Actions, it is most relevant to ensure these services are not turned off.

6. Mandatory Key Rotation – Keys will be associated to users and if those users are termed, processes must be in place to replace those keys with newer keys. Promote users to utilize Secrets Manager and make sure that it is accounted for in their workflow.

7. Limit Infrastructure provisioning Types to less expensive infrastructure in Testing/QA Environments – use IAM policies to limit users to only deploying resources for which are less expensive in a QA/Testing context.

8. Use AWS Organizations to create an Account Hierarchy with relevant Service Control Policies ( SCP’s) to limit access at the account level – Use Hierarchies, similar to Active Directory AD GPO’s, to apply organization to account groups and apply limitations at the account level. Use hierarchy.

9. In regards to using AWS Organizations, although you can define Service Control Policies at every level, it is more straight forward to apply certain limiting policies that apply to all instances at the root level and defining limitations at the account level – Since every level must allow the permission ( from Account to root ), it can get very confusing. Make it easier on yourself and deal with less abstraction by concentrating only on the root level and the account level. Its okay to define FullAdmin for every level in between.

10. Use resource groups to…group resources by Tag – this is mainly useful for splitting up costs by particular resources. Can also be used to apply automatons for these particular group of tags from System Manager.

10. Use Cost Allocation Tags – Generic tags are only usable in the Billing Explorer if they have been activated in the Cost Allocation Tag Menu. Otherwise they will now show up for Billing purposes.

11. Depending on project requirements, define a non-default KMS Key by default and enable encryption by default for each region – Allows for keys to be applied, regardless if the Admin/developer forgets to do this. Best practice is to encrypt it on a project-by-project basis.

12. Deny…User Creation, Key Creation, Attaching and Detaching IAM Policies to users, Creating Identity providers, Creating and Deleting VPC’s, Creating and Deleting Subnets, Changing IGW or Route Tables, Peering Connections to all non-admin users – The idea is to lock down networking, authentication, and sources of automation which are unintended. Imagine if a hacker had to slowly provision instances through the console…

13. Deny…creating Identity providers IAM permission – Not that they will care for permission federation after they have already accessed your account.

14. Deny…Resource Sharing – If you think they should not be allowed to share, well do make sure to make sure to send a letter to their mother.

15. Use Resource Access Manager to Share Subnets from the master account to the child account – You can lock down networking as a whole by sharing certain networks with certain accounts.

16. Using SNS Billing Alerts – This can either indicate overspending or an unusual amount of resource utilization in your instance. It Should give stakeholders an idea of how much resources really cost and if they are forecasted to go above their budget.

17. Require Tagging before instances can be ran or accessed – Use IAM policies to mandata certain tag fields be present and filled out. Otherwise, in the case of EC2, it will not run and in the case of S3, it can not be accessed.

18. IAM provides a feature called Service Last Accessed -Although you can see this in the console on a per policy basis, it would probably be better if you scripted this feature to see all the Last Service dates for all relevant services. This is supposed to help with the policy of users having the least privilege.

19. Deny…Inline Policies – It is neither scalable or obvious to audit these type of policies. Just don’t allow inline policies.

20. Have an onboarding questionaire – provision only what they think they need, and give them a month to change it again. It should at least get you going on the right track.

21. If developers are using Git, make sure they use GIT Secrets – it mainly scans for credentials and such and takes them out of any git repository.