[Discussion Post] How Do You Usually Conduct POC Batch Testing?

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 【交流贴】一般你们会如何进行 POC 跑批测试？

| username: Billmay表妹

In the community, it is often heard from community users that when selecting a new database, it is not just about choosing the appropriate database based on the scenario and whether the database can meet the business needs. After selecting some target products, they will conduct batch testing and comparison of the products.

Here are a few questions for everyone to discuss:

Question 1:
How many products do you usually compare during the selection process?

Question 2:
When the results do not meet the ideal expectations during the testing and comparison, do you optimize and rerun the batch?

Question 3:
When testing, do you follow the official configuration requirements, or do you test with configurations lower than the official requirements?

Question 4:
Which SQL queries do you generally use for performance testing and comparison?

2
| username: Myth | Original post link

Q1: How many products do you choose for comparison?
A1: About 3 or 4.

Q2: Will you run the batch again after optimization?
A2: If it doesn’t meet expectations, we will optimize it.

Q3: Will you test according to the official configuration requirements, or lower than the official configuration requirements?
A3: According to the official requirements, or even higher than the official configuration.

Q4: Which SQL do you generally use for performance testing comparison?
A4: The SQL used for benchmarking.

3
| username: 张雨齐0720 | Original post link

Question 1:
How many products do you usually compare during the selection process?
----At least 2 products for comparison.

Question 2:
When the results do not meet the ideal expectations during the testing comparison, will you optimize and rerun the batch?
----We will definitely optimize, not just the database parameter configuration, but also the application configuration and SQL performance tuning. Because the SQL written by the application may have different execution efficiencies and indexes on different databases, adaptation and optimization are necessary.

Question 3:
When testing, do you follow the official configuration requirements, or do you use a lower configuration?
----For performance testing, we use configurations far higher than the official requirements for installation and deployment. We use 96C512G, NVME SSD. For functional testing, we use configurations close to the official recommendations.

Question 4:
Which SQL do you generally use for performance testing comparison?
----We use benchmarksql for self-testing, and the official test report is mainly based on the performance results of business transactions.

4
| username: 数据小黑 | Original post link

Question 1:
How many products do you usually compare during the selection process?
We typically choose 2-3 targeted products, never more than three at a time. However, some POCs may be repeated with different products each time.

Question 2:
When the results do not meet the ideal expectations during the comparison test, do you optimize and rerun the batch?
We will optimize using known methods until we believe we have identified the bottleneck or there are no short-term optimization solutions.

Question 3:
During testing, do you follow the official configuration requirements, or do you test with lower configurations?
We perform functional tests with configurations lower than the official requirements and performance tests with configurations equal to or higher than the official requirements.

Question 4:
Which SQL scenarios do you generally use for performance testing comparisons?
Key scenarios and frequently called scenarios.

5
| username: 边城元元 | Original post link

Question 1: How many products do you usually compare during the selection process?
Depending on the business scenario and other various factors, we typically narrow it down to 2 products for in-depth testing.

Question 2: When the test results do not meet ideal expectations, do you optimize and rerun the batch?
We use known methods for optimization. If the issue cannot be resolved in a blocking manner, it is considered a test conclusion.

Question 3: During testing, do you follow the official configuration requirements, or do you test with lower configurations?
We conduct preliminary and functional tests with configurations lower than the official requirements, and performance tests with configurations close to the official requirements.

Question 4: Which SQL queries do you generally use for performance testing comparisons?
Key scenario SQL, frequently called scenario SQL, and complex SQL.