How to Build a Lightweight Middle Platform Based on TiDB and Open Source Components

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 如何基于TiDB和开源组件打造一个轻量级中台

| username: 魔礼养羊

Requirements:

  1. Data Collection: Mainly from Oracle, MySQL, the group’s data sharing and exchange platform, and a large number of standalone systems (possibly using SQLite, Access). The collection cycle is generally on a daily basis.
  2. Computation: Micro-batch processing is sufficient, OLTP data volume is very small; there are some OLTP requirements, but they do not change much; the volume of log analysis is also not large.
  3. Storage: Huawei Cloud Storage, the annual data volume should not exceed 10 million records.
  4. Governance: There are governance requirements.
  5. Shared Exchange: Need to provide shared exchange functionality.
  6. Visualization: There is an IOC platform that needs to call data indicator results from the data middle platform.

Plan:
Use TiDB for data import, storage, data warehouse, and services; use Flink for stream and batch processing; Kafka for message queue; Redis for caching; ES for log collection; Data Hub for data governance (open-sourced by LinkedIn, not Alibaba); ZK/YARN/HDFS are not emphasized.

Drawbacks:
Can TiDB achieve these functionalities? Hope to have a unified web management interface.

Only have theoretical knowledge, some CRUD experience with MySQL, hope to get guidance from everyone, please be gentle!!

2
| username: xingzhenxiang | Original post link

To build a lightweight middle platform based on TiDB and open-source components, you can follow these steps:

Architecture Design: First, analyze the business to determine the functions and service scope of the middle platform, then design the corresponding technical architecture. You can adopt a microservices architecture, dividing the middle platform into different service modules, each providing specific functions and services. At the same time, consider the design of the data storage solution. TiDB can be used as the data storage layer, and other open-source components can also be used for support.

System Implementation: Based on the architecture design, proceed with system implementation. This includes writing code, configuring the environment, and setting up the runtime platform. During implementation, pay attention to the maintainability, scalability, and stability of the code. Additionally, perform integration testing of each service module to ensure the entire middle platform operates correctly.

Deployment and Operations: Deploy the middle platform to the production environment and carry out monitoring, log management, troubleshooting, and security assurance tasks. TiDB itself provides some monitoring and tuning tools, which facilitate operations management. Additionally, other open-source tools can be used to assist in operations management.

Iteration and Optimization: As the business evolves, the middle platform needs continuous iteration and optimization. Analyze user feedback, performance metrics, and other data to tune and optimize the system. New open-source components can also be introduced to meet business needs.

In summary, building a lightweight middle platform based on TiDB and open-source components requires a deep understanding of business needs, reasonable architecture design, high-quality code implementation, standardized operations management, and continuous optimization and iteration to achieve better results.

3
| username: TiDBer_pkQ5q1l0 | Original post link

TiDB is a distributed database designed to be compatible with the MySQL protocol and support horizontal scaling. To build a lightweight middle platform based on TiDB and open-source components, you can consider the following aspects:

  1. Data Storage and Management: Use TiDB as the foundation for data storage and management to achieve distributed, high availability, and horizontal scaling features. Additionally, you can use TiDB’s distributed transaction features to ensure data consistency and reliability.
  2. Service Registration and Discovery: Use open-source components like Consul to implement service registration and discovery functions, facilitating communication and collaboration between services. This can achieve a service-oriented architecture for the middle platform, enhancing the system’s flexibility and scalability.
  3. Configuration Management: Use open-source components like Etcd to implement configuration management functions, centralizing the management of configuration information for various services, making it easier to manage and adjust configurations uniformly.
  4. API Gateway: Use an API gateway (such as Kong, Nginx, etc.) to provide service interfaces externally, enabling request forwarding, authentication, rate limiting, and other functions, while also improving the system’s security and stability.
  5. Monitoring and Logging: Use open-source components like Prometheus to implement monitoring and logging functions, allowing you to monitor and analyze the system’s operational status, and promptly identify and resolve issues.
  6. Microservice Framework: Use open-source microservice frameworks like Spring Cloud to implement service calls and collaboration, enabling the splitting and decoupling of business logic, thereby improving the system’s maintainability and scalability.

In summary, building a lightweight middle platform based on TiDB and open-source components requires the comprehensive use of multiple open-source components and consideration of their integration and collaboration. This requires a certain level of technical expertise and practical experience, as well as reasonable selection and configuration based on actual needs.

The above is chartGPT’s response!

4
| username: 魔礼养羊 | Original post link

Thank you, brother, for your reply. I will take your product planning ideas into consideration.

5
| username: 魔礼养羊 | Original post link

I think this response is quite good, but it still has significant shortcomings.
At least data governance and data sharing/exchange were not considered.
Other common components were not considered as much before, thank you very much.

6
| username: 魔礼养羊 | Original post link

Thanks to the brothers at NetEase. They gave me the idea of BI + Lakehouse + Data Governance Platform. It turns out to be the most cost-effective solution.