体验版地址 | 账密 poc/123456
首页
数据集成
元数据管理
元数据拾取
应用分析
系统菜单管理
元数据管理
数据质量
数据市场
数据标准
BI报表
数据资产
流程编排
AllData AI Studio 社区版
AllData Studio 社区版
Dlink
FlinkX
ElAdmin
Dlink+CDC+Hudi
cube-studio
ElAdmin
ElAdmin
Rancher
Hive+Doris
Dlink+FlinkCDC+Doris
DolphinScheduler
SREWorks
Doris
lowcode-engine
数据库版本为 mysql5.7 及以上版本
1.1 source install/eladmin/eladmin_alldatadc.sql
1.2 source install/eladmin/eladmin_dts.sql
1.3 source install/datax/eladmin_data_cloud.sql
1.4 source install/datax/eladmin_cloud_quartz.sql
1.5 source install/datax/eladmin_foodmart2.sql
1.6 source install/datax/eladmin_robot.sql
config 文件夹下的配置文件,修改 redis,mysql 和 rabbitmq 的配置信息
cd install/datax
mvn install:install-file -DgroupId=com.aspose -DartifactId=aspose-words -Dversion=20.3 -Dpackaging=jar -Dfile=aspose-words-20.3.jar
获取安装包build/eladmin-release-2.6.tar.gz
上传服务器解压
5.1 必须启动、并且顺序启动
eureka->config->gateway
5.2 按需启动
cd install/16gmaster譬如启动元数据管理
sh
install/16gmaster/data-metadata-service.shtail -100f
install/16gmaster/data-metadata-service.log5.2 按需启动
cd install/16gdata按需启动相关服务
5.3 按需启动
cd install/16gslave按需启动相关服务
6.1 启动
sh install/16gmaster/eladmin-system.sh6.2 部署
Eladmin前端source /etc/profile
cd $(dirname 0ドル)
source /root/.bashrc && nvm use v10.15.3
nohup npm run dev &
6.3 访问
Eladmin页面用户名:admin 密码:123456
## 知识图谱(Knowledge Graph)
从知识抽取的内容上, 又可以分为实体抽取, 属性抽取, 关系抽取, 事件抽取:
实体抽取指从数据源中检测到可命名的实体, 并将它们分类到已建模的类型中, 例如人, 组织, 地点, 时间等等;
属性抽取是识别出命名实体的具体属性;
关系抽取是识别出实体与实体之间的关系, 例如从句子"著名歌手周杰伦的妻子昆凌"中识别出"周杰伦"与"昆凌"之间的夫妻关系;
事件抽取是识别出命名实体相关的事件信息, 例如"周杰伦"与"昆凌"结婚就是一个事件
可以看出实体抽取, 属性抽取, 关系抽取是抽取我们在知识建模中定义的拓扑结构部分数据,
事件抽取是事件建模相关数据的抽取, 所以在领域知识图谱建设中, 也需要包括数据准备域的抽取方式, 处置域的数据抽取方式
知 识 验 证
从各种不同数据源抽取的知识, 并不一定是有效的知识, 必须进行知识的验证, 将有效的, 正确的知识进入知识库造成知识不准确的原因,
通常是原始数据存在错误, 术语存在二义性, 知识冲突等等, 例如前面提到的"1#"压水堆, "1号"压水堆, "一号"压水堆这三个词对应一个实体,
如果在抽取中没有合理定义规则, 这就需要在知识验证阶段得到处理, 以便形成闭环
1, 知识图谱建设
1.1 人工数据标注工具: https://github.com/doccano/doccano
1.2 自动标注+知识抽取: https://github.com/zjunlp/DeepKE
@Test
void testCreateDatabase() {
sql("create database db1").ok("CREATE DATABASE `DB1`");
sql("create database db1 comment 'comment db1' location '/path/to/db1'")
.ok(
"CREATE DATABASE `DB1`\n"
+ "COMMENT 'comment db1'\n"
+ "LOCATION '/path/to/db1'");
sql("create database db1 with dbproperties ('k1'='v1','k2'='v2')")
.ok(
"CREATE DATABASE `DB1` WITH DBPROPERTIES (\n"
+ " 'k1' = 'v1',\n"
+ " 'k2' = 'v2'\n"
+ ")");
}
测试FlinkHiveSqlParser Passed
参考Resource/FlinkDDLSQL.sql
CREATE TABLE data_gen (
amount BIGINT
) WITH (
'connector' = 'datagen',
'rows-per-second' = '1',
'number-of-rows' = '3',
'fields.amount.kind' = 'random',
'fields.amount.min' = '10',
'fields.amount.max' = '11');
CREATE TABLE mysql_sink (
amount BIGINT,
PRIMARY KEY (amount) NOT ENFORCED
) WITH (
'connector' = 'jdbc',
'url' = 'jdbc:mysql://localhost:3306/test_db',
'table-name' = 'test_table',
'username' = 'root',
'password' = '123456',
'lookup.cache.max-rows' = '5000',
'lookup.cache.ttl' = '10min'
);
INSERT INTO mysql_sink SELECT amount as amount FROM data_gen;
获取结果
1、Flink血缘构建结果-表:
[LineageTable{id='4', name='data_gen', columns=[LineageColumn{name='amount', title='amount'}]},
LineageTable{id='6', name='mysql_sink', columns=[LineageColumn{name='amount', title='amount'}]}]
表ID: 4
表Namedata_gen
表ID: 4
表Namedata_gen
表-列LineageColumn{name='amount', title='amount'}
表ID: 6
表Namemysql_sink
表ID: 6
表Namemysql_sink
表-列LineageColumn{name='amount', title='amount'}
2、Flink血缘构建结果-边:
[LineageRelation{id='1', srcTableId='4', tgtTableId='6', srcTableColName='amount', tgtTableColName='amount'}]
表-边: LineageRelation{id='1', srcTableId='4', tgtTableId='6', srcTableColName='amount', tgtTableColName='amount'}
1、BUSINESS FOR ALL DATA PLATFORM 商业项目
2、BUSINESS FOR ALL DATA PLATFORM 计算引擎
3、DEVOPS FOR ALL DATA PLATFORM 运维引擎
4、DATA GOVERN FOR ALL DATA PLATFORM 数据治理引擎
5、DATA Integrate FOR ALL DATA PLATFORM 数据集成引擎
6、AI FOR ALL DATA PLATFORM 人工智能引擎
7、DATA ODS FOR ALL DATA PLATFORM 数据采集引擎
8、OLAP FOR ALL DATA PLATFORM OLAP查询引擎
9、OPTIMIZE FOR ALL DATA PLATFORM 性能优化引擎
10、DATABASES FOR ALL DATA PLATFORM 分布式存储引擎
set execution.checkpointing.interval=15sec;
CREATE CATALOG alldata_catalog WITH (
'type'='table-store',
'warehouse'='file:/tmp/table_store'
);
USE CATALOG alldata_catalog;
CREATE TABLE word_count (
word STRING PRIMARY KEY NOT ENFORCED, cnt BIGINT);
CREATE TEMPORARY TABLE word_table (
word STRING) WITH (
'connector' = 'datagen', 'fields.word.length' = '1');
INSERT INTO word_count SELECT word, COUNT(*) FROM word_table GROUP BY word;
-- POC Test OLAP QUERY
SET sql-client.execution.result-mode = 'tableau';
RESET execution.checkpointing.interval;
SET execution.runtime-mode = 'batch';
SELECT * FROM word_count;
-- POC Test Stream QUERY
-- SET execution.runtime-mode = 'streaming';
-- SELECT
interval, COUNT(*) AS interval_cnt FROM-- (SELECT cnt / 10000 AS
intervalFROM word_count) GROUP BYinterval;
### 2、Dlink启动并运行成功
### 3、OLAP查询
4.1 Stream Read 1
> 4.2 Stream Read 2
| Component | Description | Important Composition |
|---|---|---|
| ai | AI STUDIO FOR ALL DATA PLATFORM artificial intelligence engine | 人工智能引擎 |
| assembly | WHOLE PACKAGE BUILD FOR ALL DATA PLATFORM assembly engine | 整包构建引擎 |
| buried | BURIED FOR ALL DATA PLATFORM data acquisition engine | 埋点解决方案 |
| buried-trade | BURIED TRADE FOR ALL DATA PLATFORM commerce engine | 商业系统 |
| cluster | DATA SRE FOR ALL DATA PLATFORM OLAP query engine | 智能大数据运维引擎 |
| crawlerlab | CRAWLER PLATFORM FOR ALL DATA PLATFORM commerce engine | 爬虫引擎系统 |
| document | DOCUMENT FOR ALL DATA PLATFORM OLAP query engine | 官方文档 |
| dts | DTS FOR ALL DATA PLATFORM DATA DTS engine | 数据集成引擎 |
| fs | DATA STORAGE FOR ALL DATA PLATFORM DATA STORAGE engine | 大数据存储引擎 |
| govern | DATA GOVERN FOR ALL DATA PLATFORM Data Governance Engine | 数据治理引擎 |
| iot | IOT FOR ALL DATA PLATFORM Data Governance Engine | 云原生IOT开发框架 |
| knowledge | KNOWLEDGE GRAPH FOR ALL DATA PLATFORM Data Task Engine | 知识图谱引擎 |
| lakehouse | ONE LAKE FOR ALL DATA PLATFORM ONE LAKE engine | 数据湖引擎 |
| market | MARKET FOR ALL DATA PLATFORM MARKET engine | 数据实验场引擎 |
| olap | OLAP FOR ALL DATA PLATFORM OLAP query engine | 混合OLAP查询引擎 |
| studio | ONE HUB FOR ALL DATA PLATFORM ONE HUB Engine | AllData总部前后端解决方案 |
| trade | TRADE FOR ALL DATA PLATFORM TRADE Engine | TRADE引擎 |
| wiki | WIKI FOR ALL DATA PLATFORM WIKI Engine | AllData知识库 |
| alldata | AllData社区项目通过二开大数据生态组件,以及大数据采集、大数据存储、大数据计算、大数据开发来建设一站式大数据平台 | Github一站式开源大数据平台AllData社区项目 |
1、AllData前端解决方案
studio/eladmin-web2、AllData后端解决方案
studio/eladmin3、多租户运维平台前端
studio/tenant4、多租户运维平台前端
studio/tenantBack
