2022数据中台建设全套方案

本方案  本方案通过对客户大数据应用平台服务需求的理解，根据建设目标、设计原则的多方面考虑，建议采用星环科技Transwarp Data Hub（TDH）大数据基础平台的架构方案，基于Transwarp Operating System（简称TOS）云平台方式部署构建。    通过建立大数据集成平台、大数据计算平台、大数据开发平台及大数据运维平台来满足客户大数据应用平台服务建设的要求。1.1.1. 大数据平台-TDH    星环大数据平台Transwarp Data Hub（简称TDH）基于开源Spark技术，从查询引擎、计算框架、存储引擎和资源调度等方面做了性能的优化，相较于同类产品，提供更好的性能。改进的分布式计算引擎Inceptor，解决了开源Spark的稳定性问题，并且已经在众多成功案例中经历了的考验。同时Ipcetor大幅提高Spark计算性能，是开源的2-10倍。TDH Inceptor极大提高了Spark功能和性能的稳定性，可以7*24小时在企业的生产环境运行，并能在TB级规模数据上高效进行各种稳定的统计分析。星环大数据平台TDH采用基于Hadoop的数据平台架构，海量数据查询分析服务集群既可以处理结构化的数据，也可以处理非结构化、半结构化的数据，满足配置、日志、网页、音视频、社交网络等多源异构数据的加载和存储，提供数据查询、全文检索、数据离线批处理分析、交互式分析、图分析、数据挖掘、机器学习等多种数据处理模式。同时，基于平台提供的实时流处理集群，可以满足实时数据研判分析服务的需求。整个平台提供完整的多租户功能，对于计算资源与存储资源以及数据访问资源进行统一控制管理，对于计算资源进行高效的调度管理与使用控制；对于存储资源进行配额管理；对于数据访问权限，进行严格的权限管理。在安装、配置、监控、告警方面，通过统一的Transwarp Manager进行运维管理。星环大数据平台TDH应用范围覆盖各种规模和不同数据量的企业，通过内存计算、高效索引、执行优化和高度容错的技术，使得一个平台能够处理10GB到100PB的数据，并且在每个数量级上，都能比现有技术提供更快的性能；企业客户不再需要混合架构，TDH可以伴随企业客户的数据增长，动态不停机扩容，避免MPP或混合架构数据迁移的棘手问题。

This proposal, based on an in-depth understanding of the service requirements of the customer's big data application platform and considering multiple factors including construction objectives and design principles, recommends adopting the architecture solution based on Transwarp Data Hub (TDH), the big data basic platform developed by Transwarp Technologies, and deploying it on the Transwarp Operating System (TOS for short) cloud platform.

This solution meets the customer's requirements for building a big data application platform service by establishing a big data integration platform, big data computing platform, big data development platform, and big data operation and maintenance platform.

1.1.1. Big Data Platform - TDH
Transwarp Data Hub (TDH), the big data platform developed by Transwarp Technologies, is built on open-source Spark technology. It has optimized performance in terms of query engine, computing framework, storage engine, resource scheduling and other aspects, delivering better performance than competing products. The improved distributed computing engine Inceptor resolves the stability issues of open-source Spark, and has been validated in numerous successful use cases. Meanwhile, Inceptor significantly boosts the computing performance of Spark, reaching 2 to 10 times that of the open-source version. TDH Inceptor greatly improves the stability of Spark's functions and performance, enabling 24/7 operation in enterprise production environments and efficient and stable statistical analysis on terabyte-scale data.

TDH adopts a Hadoop-based data platform architecture. Its massive data query and analysis service cluster can handle structured, unstructured and semi-structured data. It supports the loading and storage of multi-source heterogeneous data such as configuration files, logs, web pages, audio and video, and social network data, and provides various data processing modes including data query, full-text search, offline batch analysis, interactive analysis, graph analysis, data mining and machine learning. Meanwhile, the real-time stream processing cluster provided by the platform meets the requirements of real-time data analysis and evaluation services.

The entire platform offers complete multi-tenant functions, with unified control and management over computing resources, storage resources and data access resources. It implements efficient scheduling management and usage control for computing resources, quota management for storage resources, and strict permission management for data access rights. It conducts operation and maintenance management via the unified Transwarp Manager for installation, configuration, monitoring and alerting.

The application scope of Transwarp TDH covers enterprises of all sizes and data volumes. Leveraging technologies such as in-memory computing, efficient indexing, execution optimization and high fault tolerance, a single TDH platform can process data ranging from 10GB to 100PB, delivering faster performance than existing technologies at every data scale. Enterprise customers no longer need hybrid architectures: TDH can dynamically scale out without downtime as the customer's data grows, avoiding the thorny issues of data migration associated with MPP or hybrid architectures.

搜集汇总

背景与挑战

背景概述

该数据集提供了2022年数据中台建设的全套解决方案，基于星环科技的TDH平台，涵盖大数据集成、计算、开发和运维等多个方面，适用于不同规模企业的数据处理需求。