Blink计算引擎蒋晓伟xiaowei.jxw@alibaba-inc.com个人简介•蒋晓伟•2014−−现在阿里巴巴•2010−−2014脸书•2002−−2010微软•2000−−2002Stratify提纲•背景和用例•什么是Blink?•Blink的改进•现状和计划用例－搜索文档的创建和更新•开发效率•全量增量一套代码•高层次API•一致性•至少一次•恰好一次•低延迟•亚秒级•成本•高吞吐FilterItemFilterCategorySellerJoinSearchEngineHBaseResultUICDataSourceItemSellerHBaseExportSync…...全量增量一体化的抽象•结果表＝物化视图•全量–索引的创建和重建•增量–索引的维护CREATEMATERIALIZEDVIEWResultASSELECT*,UDF(a,b,c)FROMItemJOINSellerONItem.uid=Seller.idItemSellerResultJoinUDF流和表的对偶性流的等价性:两个流等价<==>它们产生相同的正则历史表wordcountHello3World1Bark1wordcountHello1World1Hello2Bark1Hello3流历史表MaterializeChangelogWordCount例子(批处理)timeword1Hello2World3Hello4Hello5BarkSELECTword,count(*)FROMWordsGROUPBYwordWordCountWordsSELECTsum(count)FROMWordCounttotal5TotalwordcountHello3World1Bark1WordCount例子(流处理)timeword1Hello2World3Hello4Hello5BarkSELECTword,count(*)FROMWordsGROUPBYwordWordCountWordsSELECTsum(count)FROMWordCountTotalwordcountHello1World1Hello2Hello3Bark1wordcountHello3World1Bark1total12478Wrong!WordCount例子(Retraction)timeword1Hello2World3Hello4Hello5BarkSELECTword,count(*)FROMWordsGROUPBYwordWordCountWordsSELECTsum(count)FROMWordCountTotalwordcountHello1World1Hello1Hello2Hello2Hello3Bark1wordcountHello3World1Bark1total1213145total5ApacheSpark•Spark•高层次的API－开发效率•Tungsten－很多性能优化•较活跃的生态•SparkStreaming•Micro-batching•很难做到亚秒级延迟流计算批处理ApacheFlink•流和批的一体化计算引擎•开发效率：高层次API•一致性：有状态的计算•低延迟•高吞吐Flink生态什么是Blink?•Blink–阿里巴巴基于Flink开发的计算引擎•批和流一体化的完备的TableAPI•和FlinkAPI以及生态兼容的重新开发的RuntimeBlink的TableAPI•原则–流和批的一体化处理•功能•UDF/UDTF/UDAGG•双流Join•Aggregation(min,max,avg,sum,count,distinct_count)•Windowing(time_window,count_window)•撤回（Retraction）Blink的Runtime•和YARN的原生态整合•Checkpoint和状态管理的优化•容错性和高可用性•动态伸缩•稳定性和可运维性的改进FlinkonYARNBlinkYARN的原生态整合Chandy-LamportCheckpointBlink的Checkpoint和状态管理BlinkWorker的错误恢复BlinkMaster的高可用性Blink的动态伸缩Blink监控JobVertexNumber:[CPU,Memory]*ParallelismInQueueTPSOutQueueLatencyDelayCPUMemoryTaskMetricsRunningTasks现状和计划•已在上千台机器的集群上线•支持搜索和推荐等核心业务•集团内外的浓厚兴趣•NetFlix,Airbnb,Uber,Facebook•反馈给社区Thankyou!
