Are there any existing metrics, computations and/or calculations to determine the right infrastructure/technology capacity requirements for different Big Data implementation projects? We want to identify the right infrastructure specifications for different clients, depending on different factors and needs. What are the inputs to be able to define this?
We are currently trying to determine the right infrastructure requirements for a client. However, I was wondering whether there are existing calculations, concepts, procedures, resources, etc. in order to make this task more “scientific”. We wish to avoid inaccurate estimates, trial-and-error, and unnecessary adjustments on infrastructure.
Here are the considerations that I believe are possible factors for a “calculation”, if any exists. But can you please add more?
1) Size of total data to ingest
2) What components are installed (spark, parquet, hbase, etc)
3) Size of chunk of data for delta changes
4) Real time or batch processsing
5) Others
In the end, I’m looking for any well-defined formula for determining the “right” tech for different projects.
Appreciate any inputs and resources. Thanks in advance!
Update:
I found this Question in StackOverlfow:
However, I was wondering if there are more better-defined ways of identifying the right infrastructure capacity requirements for Big Data. One answer mentioned that it is best to test to figure out what is best. But that is what we want to minimise if we had a defined formula for it.