欢迎大家赞助一杯啤酒🍺 我们准备了下酒菜:Formal mathematics/Isabelle/ML, Formal verification/Coq/ACL2, C++/F#/Lisp
Apache Spark
来自开放百科 - 灰狐
(版本间的差异)
小 (→链接) |
小 (→图集) |
||
(未显示1个用户的48个中间版本) | |||
第1行: | 第1行: | ||
Apache Spark:新一代大数据解决方案 | Apache Spark:新一代大数据解决方案 | ||
− | Spark 是用[[Scala]]语言编写的一套分布式内存计算系统,它的核心抽象模型是 RDD (Resilient Distributed Dataset,弹性分布式数据集),围绕 RDD 构建了一系列分布式 API,可以直接对数据集进行分布式处理。 | + | ==简介== |
+ | Spark 是用[[Scala]]和[[Java]]语言编写的一套分布式内存计算系统,它的核心抽象模型是 RDD (Resilient Distributed Dataset,弹性分布式数据集),围绕 RDD 构建了一系列分布式 API,可以直接对数据集进行分布式处理。 | ||
相对于 [[MapReduce]] 上的批量计算、迭代计算,以及基于 [[Apache Hive]] 的 SQL 查询,Spark 可以带来一到两个数量级的性能提升。 | 相对于 [[MapReduce]] 上的批量计算、迭代计算,以及基于 [[Apache Hive]] 的 SQL 查询,Spark 可以带来一到两个数量级的性能提升。 | ||
Spark在广告领域有很多的成功应用。 | Spark在广告领域有很多的成功应用。 | ||
+ | |||
+ | ==版本== | ||
+ | *2.3 | ||
+ | *[https://github.com/apache/spark/tree/branch-2.2 2.2] | ||
+ | *[https://github.com/apache/spark/tree/branch-2.1 2.1] | ||
+ | *[https://github.com/apache/spark/tree/branch-2.0 2.0] | ||
==Apache Hive== | ==Apache Hive== | ||
[[Hive on Spark]] | [[Hive on Spark]] | ||
+ | |||
+ | ==Spark on HBase== | ||
+ | *使用 [[HDFS]] 内存层实现 RDD 共享 | ||
+ | *[http://spark-packages.org/package/Huawei-Spark/Spark-SQL-on-HBase Spark SQL on HBase]: Spark SQL/DataFrame access to NoSQL data in [[Apache HBase]] | ||
+ | |||
+ | ==Spark on YARN== | ||
+ | 通过Spark on YARN的方式与[[Apache Hadoop]]方便地共享集群的存储功能和计算资源。 | ||
+ | |||
+ | ==[[Apache Cassandra|Cassandra]]== | ||
+ | *[http://www.datastax.com/dev/blog/zen-art-spark-maintenance Zen and the Art of Spark Maintenance] | ||
+ | *[https://github.com/datastax/spark-cassandra-connector DataStax Spark Cassandra Connector] | ||
+ | *[https://tobert.github.io/post/2014-07-15-installing-cassandra-spark-stack.html Installing the Cassandra / Spark OSS Stack] | ||
+ | |||
+ | ==[[elasticsearch]]== | ||
+ | *[https://www.elastic.co/guide/en/elasticsearch/hadoop/master/spark.html ElasticSearch Spark Integration] | ||
+ | |||
+ | ==REST API== | ||
+ | Spark 1.4引入REST API | ||
+ | http://localhost:4040/api/v1/applications | ||
+ | |||
+ | ==机器学习== | ||
+ | *[http://mlbase.org/ MLbase] | ||
+ | *[[Apache Mahout]]使用Spark作为后端 | ||
+ | *[https://github.com/ariskk/distributedWekaSpark Weka on Spark] | ||
+ | *[https://github.com/amplab/keystone KeystoneML] | ||
+ | *[[SparkNet]] | ||
+ | *[https://github.com/gingsmith/cocoa CoCoA] | ||
+ | *[https://github.com/zhangyuc/splash Splash Project for parallel stochastic learning] | ||
+ | *[[Deeplearning4j]] | ||
+ | *[[H2O]] Spark | ||
+ | *[https://github.com/yahoo/TensorFlowOnSpark TensorFlowOnSpark] | ||
+ | *Yahoo [https://github.com/yahoo/CaffeOnSpark CaffeOnSpark] | ||
+ | *[https://arimo.com/machine-learning/deep-learning/2016/arimo-distributed-tensorflow-on-spark/ Arimo Tensorflow On Spark] | ||
+ | |||
+ | ==项目== | ||
+ | [https://github.com/awesome-spark/awesome-spark Awesome Spark] [[文件:awesome.png]] | ||
+ | *[https://cwiki.apache.org/confluence/display/SPARK/Supplemental+Spark+Projects Spark的相关项目和生态系统: Supplemental Spark Projects] | ||
+ | *[https://github.com/spark-jobserver/spark-jobserver spark-jobserver] | ||
+ | *[https://github.com/amplab-extras/SparkR-pkg SparkR] | ||
+ | *[[Apache Mesos]] | ||
+ | *[[Alluxio]] | ||
+ | *[https://github.com/tuplejump/FiloDB FiloDB] | ||
+ | *[[Apache Zeppelin]], [https://github.com/andypetrella/spark-notebook/ Spark Notebook] | ||
+ | *[https://github.com/SnappyDataInc/snappydata SnappyData: OLTP + OLAP Database built on Apache Spark] | ||
+ | *[http://blinkdb.org/ BlinkDB] | ||
+ | *[https://github.com/adobe-research/spindle Spindle] | ||
+ | *[http://simin.me/projects/spatialspark/ SpatialSpark] | ||
+ | *[https://toree.apache.org Apache Toree] | ||
+ | *[[Apache SystemML]] | ||
+ | *[[Oryx]] | ||
+ | *[https://github.com/SnappyDataInc/snappydata SnappyData] OLTP + OLAP Database built on Apache Spark [http://docs.huihoo.com/apache/spark/summit/2016/efficient-state-management-with-spark-20-and-scale-out-databases.pdf Efficient State Management With Spark 2.0 And Scale-Out Databases] | ||
+ | *[http://livy.io/ Livy] an Open Source REST Service for Spark | ||
+ | |||
+ | ==服务商== | ||
+ | *[http://databricks.com/ Spark背后的商业公司:Databricks],同时提供Spark服务提供商Certified Spark Distribution官方认证。 | ||
+ | *[http://www.stratio.com Stratio Platform]: The first "pure Spark" platform with 50% fewer components and operational complexity. | ||
+ | *IBM的思路是将Spark视为数据分析的操作系统。 | ||
+ | |||
+ | ==用户== | ||
+ | *[https://cwiki.apache.org/confluence/display/SPARK/Powered+By+Spark Powered By Spark] | ||
+ | *[https://zhuanlan.zhihu.com/p/27538270 60 TB 数据:Facebook 是如何大规模使用 Apache Spark 的] | ||
+ | |||
+ | ==课程== | ||
+ | *[http://docs.huihoo.com/big-data/introduction-to-big-data-with-apache-spark/ Introduction to Big Data with Apache Spark] | ||
+ | *[http://docs.huihoo.com/machine-learning/scalable-machine-learning/ Scalable Machine Learning] | ||
==文档== | ==文档== | ||
+ | *[http://docs.huihoo.com/apache/spark/Matrix-Computations-and-Neural-Networks-in-Spark.pdf Spark上的矩阵计算和神经网络] | ||
*[http://docs.huihoo.com/apache/apachecon/us2015/Hive-Now-Sparks.pdf Hive Now Sparks] | *[http://docs.huihoo.com/apache/apachecon/us2015/Hive-Now-Sparks.pdf Hive Now Sparks] | ||
*[http://docs.huihoo.com/apache/apachecon/us2015/Going-Deep-with-Spark-Streaming.pdf Going Deep with Spark Streaming] | *[http://docs.huihoo.com/apache/apachecon/us2015/Going-Deep-with-Spark-Streaming.pdf Going Deep with Spark Streaming] | ||
第17行: | 第90行: | ||
*[http://docs.huihoo.com/apache/spark/summit/east2015/SSE15-11-Delivering-Meaning-At-High-Velocity-with-Spark-Streaming-Cassandra-Kafka-and-Akka.pdf Delivering Meaning In NearReal Time At High Velocity & Massive Scale] | *[http://docs.huihoo.com/apache/spark/summit/east2015/SSE15-11-Delivering-Meaning-At-High-Velocity-with-Spark-Streaming-Cassandra-Kafka-and-Akka.pdf Delivering Meaning In NearReal Time At High Velocity & Massive Scale] | ||
*[http://docs.huihoo.com/apache/apachecon/us2015/Streaming-OODT-Combining-Apache-Spark-Power-with-Apache-OODT.pdf Streaming OODT: Combining Apache Spark's Power with Apache OODT] | *[http://docs.huihoo.com/apache/apachecon/us2015/Streaming-OODT-Combining-Apache-Spark-Power-with-Apache-OODT.pdf Streaming OODT: Combining Apache Spark's Power with Apache OODT] | ||
+ | *[http://docs.huihoo.com/infoq/qconshanghai/2015/%e5%9f%ba%e4%ba%8e%e5%a4%a7%e6%95%b0%e6%8d%ae%e7%9a%84%e6%9c%ba%e5%99%a8%e5%ad%a6%e4%b9%a0%e6%8a%80%e6%9c%af/QCon%e4%b8%8a%e6%b5%b72015-%e5%9f%ba%e4%ba%8e%e6%9c%ba%e5%99%a8%e5%ad%a6%e4%b9%a0%e7%9a%84%e9%93%b6%e8%a1%8c%e5%8d%a1%e6%b6%88%e8%b4%b9%e6%95%b0%e6%8d%ae%e9%a2%84%e6%b5%8b%e4%b8%8e%e6%8e%a8%e8%8d%90-%e6%a2%81%e5%a0%b0%e6%b3%a2.pdf Machine learning in finance using Spark ML pipeline] | ||
+ | *[http://docs.huihoo.com/infoq/qconshanghai/2015/%e5%9f%ba%e4%ba%8e%e5%a4%a7%e6%95%b0%e6%8d%ae%e7%9a%84%e6%9c%ba%e5%99%a8%e5%ad%a6%e4%b9%a0%e6%8a%80%e6%9c%af/QCon%e4%b8%8a%e6%b5%b72015-%e5%9c%a8Spark%e4%b8%8a%e6%9e%84%e5%bb%ba%e7%a1%ac%e4%bb%b6%e5%8a%a0%e9%80%9f%e7%9a%84%e5%88%86%e5%b8%83%e5%bc%8f%e7%a5%9e%e7%bb%8f%e7%bd%91%e7%bb%9c%e6%9e%b6%e6%9e%84-%e7%8e%8b%e5%a5%95%e6%81%92%e3%80%81%e9%bb%84%e6%99%9f%e7%9b%9b.pdf 在Spark上构建分布式神经网络] | ||
==图集== | ==图集== | ||
<gallery> | <gallery> | ||
+ | image:spark-runs-everywhere.png|Spark | ||
image:spark-streaming-architecture.png|Streaming | image:spark-streaming-architecture.png|Streaming | ||
image:Spark-Streaming-Cassandra-Kafka-and-Akka.png|堆栈 | image:Spark-Streaming-Cassandra-Kafka-and-Akka.png|堆栈 | ||
+ | image:Spark-and-Map-Reduce-Differences.png|Spark和MapReduce | ||
+ | image:apache-spark-cluster.png|Spark集群 | ||
+ | image:Berkeley-Data-Analytics-Stack.png|伯克利数据分析堆栈BDAS | ||
+ | image:baidu-spark-one.png|百度Spark One | ||
+ | image:hadoop-vs-spark.png|Hadoop vs Spark | ||
+ | image:Spark-Streaming.png|Spark Streaming | ||
</gallery> | </gallery> | ||
第29行: | 第111行: | ||
*[https://amplab.cs.berkeley.edu/ Spark发源地:AMPLab] | *[https://amplab.cs.berkeley.edu/ Spark发源地:AMPLab] | ||
*[http://docs.huihoo.com/big-data/ampcamp/ UC Berkeley Big Data AMP Camp 大会资料] | *[http://docs.huihoo.com/big-data/ampcamp/ UC Berkeley Big Data AMP Camp 大会资料] | ||
− | |||
*[http://docs.huihoo.com/apache/spark Spark开放文档] | *[http://docs.huihoo.com/apache/spark Spark开放文档] | ||
*[http://spark-summit.org/ Spark Summit] | *[http://spark-summit.org/ Spark Summit] | ||
+ | *[http://spark-packages.org/ Spark Packages] | ||
*[http://mp.weixin.qq.com/s?__biz=MzA3MDQ4MzQzMg==&mid=205137135&idx=1&sn=24de5f0a11d8a4f3bb40854717c1fa42&scene=4&uin=MTEyNDM1NzgxMA%3D%3D&key=88c7ba82076acca7edd27801f1257cdb75ade13b909dd6ed6c053bc893f6cc1ca31174495f0c63c423df1953fbdbadd7&devicetype=android-17&version=25030050&lang=zh_CN 腾讯大数据之计算新贵Spark,广点通是最早使用Spark的应用之一] | *[http://mp.weixin.qq.com/s?__biz=MzA3MDQ4MzQzMg==&mid=205137135&idx=1&sn=24de5f0a11d8a4f3bb40854717c1fa42&scene=4&uin=MTEyNDM1NzgxMA%3D%3D&key=88c7ba82076acca7edd27801f1257cdb75ade13b909dd6ed6c053bc893f6cc1ca31174495f0c63c423df1953fbdbadd7&devicetype=android-17&version=25030050&lang=zh_CN 腾讯大数据之计算新贵Spark,广点通是最早使用Spark的应用之一] | ||
*[http://www.biaodianfu.com/spark-tdw.html Spark在腾讯数据仓库TDW的应用] | *[http://www.biaodianfu.com/spark-tdw.html Spark在腾讯数据仓库TDW的应用] | ||
第39行: | 第121行: | ||
*[http://www.csdn.net/article/2015-04-30/2824594-spark-summit-china-2015 七牛技术总监陈超:记Spark Summit China 2015] | *[http://www.csdn.net/article/2015-04-30/2824594-spark-summit-china-2015 七牛技术总监陈超:记Spark Summit China 2015] | ||
*[http://www.csdn.net/article/2015-07-30/2825342 七牛是如何搞定每天500亿条日志的] | *[http://www.csdn.net/article/2015-07-30/2825342 七牛是如何搞定每天500亿条日志的] | ||
+ | *[http://datascienceassn.org/content/39-machine-learning-libraries-spark-categorized 39 Machine Learning Libraries for Spark] | ||
− | |||
[[category:big data]] | [[category:big data]] | ||
+ | [[category:data science]] | ||
[[category:hadoop]] | [[category:hadoop]] | ||
[[category:scala]] | [[category:scala]] | ||
+ | [[category:apache]] | ||
+ | [[category:hortonworks]] | ||
+ | [[category:huihoo]] |
2022年4月20日 (三) 00:10的最后版本
Apache Spark:新一代大数据解决方案
目录 |
[编辑] 简介
Spark 是用Scala和Java语言编写的一套分布式内存计算系统,它的核心抽象模型是 RDD (Resilient Distributed Dataset,弹性分布式数据集),围绕 RDD 构建了一系列分布式 API,可以直接对数据集进行分布式处理。
相对于 MapReduce 上的批量计算、迭代计算,以及基于 Apache Hive 的 SQL 查询,Spark 可以带来一到两个数量级的性能提升。
Spark在广告领域有很多的成功应用。
[编辑] 版本
[编辑] Apache Hive
[编辑] Spark on HBase
- 使用 HDFS 内存层实现 RDD 共享
- Spark SQL on HBase: Spark SQL/DataFrame access to NoSQL data in Apache HBase
[编辑] Spark on YARN
通过Spark on YARN的方式与Apache Hadoop方便地共享集群的存储功能和计算资源。
[编辑] Cassandra
- Zen and the Art of Spark Maintenance
- DataStax Spark Cassandra Connector
- Installing the Cassandra / Spark OSS Stack
[编辑] elasticsearch
[编辑] REST API
Spark 1.4引入REST API
http://localhost:4040/api/v1/applications
[编辑] 机器学习
- MLbase
- Apache Mahout使用Spark作为后端
- Weka on Spark
- KeystoneML
- SparkNet
- CoCoA
- Splash Project for parallel stochastic learning
- Deeplearning4j
- H2O Spark
- TensorFlowOnSpark
- Yahoo CaffeOnSpark
- Arimo Tensorflow On Spark
[编辑] 项目
- Spark的相关项目和生态系统: Supplemental Spark Projects
- spark-jobserver
- SparkR
- Apache Mesos
- Alluxio
- FiloDB
- Apache Zeppelin, Spark Notebook
- SnappyData: OLTP + OLAP Database built on Apache Spark
- BlinkDB
- Spindle
- SpatialSpark
- Apache Toree
- Apache SystemML
- Oryx
- SnappyData OLTP + OLAP Database built on Apache Spark Efficient State Management With Spark 2.0 And Scale-Out Databases
- Livy an Open Source REST Service for Spark
[编辑] 服务商
- Spark背后的商业公司:Databricks,同时提供Spark服务提供商Certified Spark Distribution官方认证。
- Stratio Platform: The first "pure Spark" platform with 50% fewer components and operational complexity.
- IBM的思路是将Spark视为数据分析的操作系统。
[编辑] 用户
[编辑] 课程
[编辑] 文档
- Spark上的矩阵计算和神经网络
- Hive Now Sparks
- Going Deep with Spark Streaming
- Significantly Speed up real world big data Applications using Apache Spark
- Finding Shoe Stores in >100k Merchants: Using Spark to Group All Things
- Delivering Meaning In NearReal Time At High Velocity & Massive Scale
- Streaming OODT: Combining Apache Spark's Power with Apache OODT
- Machine learning in finance using Spark ML pipeline
- 在Spark上构建分布式神经网络
[编辑] 图集
[编辑] 链接
- Apache Spark官网
- Spark @ GitHub
- Spark发源地:AMPLab
- UC Berkeley Big Data AMP Camp 大会资料
- Spark开放文档
- Spark Summit
- Spark Packages
- 腾讯大数据之计算新贵Spark,广点通是最早使用Spark的应用之一
- Spark在腾讯数据仓库TDW的应用
- 《Spark快速数据处理》作者 Holden Karau 曾就职于谷歌、亚马逊、微软和Foursquare等公司,对开源情有独钟,参与了许多开源项目,如Linux内核无线驱动、Android程序监控、搜索引擎等,对存储系统、推荐系统、搜索分类等都有深入研究。
- CSDN.NET Spark技术社区
- 理解Spark的核心Resilient Distributed Datasets(RDD)
- 七牛技术总监陈超:记Spark Summit China 2015
- 七牛是如何搞定每天500亿条日志的
- 39 Machine Learning Libraries for Spark
分享您的观点