Fast OLAP query execution in main memory on large data in a
cluster
M. Weidner, J. Dees, и P. Sanders. Big Data, 2013 IEEE International Conference on, стр. 518--524. (октября 2013)
Аннотация
Main memory column-stores have proven to be efficient for
processing analytical queries. Still, there has been little work
in the context of clusters. Using only a single machine poses
several restrictions: Processing power and data volume are
bounded to the number of cores and main memory fitting on one
tightly coupled system. To enable the processing of larger data
sets, switching to a cluster becomes necessary. In this work, we
explore techniques for efficient execution of analytical SQL
queries on large amounts of data in a parallel database cluster
while making maximal use of the available hardware. This
includes precompiled query plans for efficient CPU utilization,
full parallelization on single nodes and across the cluster, and
efficient inter-node communication. We implement all features in
a prototype for running a subset of TPC-H benchmark queries. We
evaluate our implementation in a 128 node cluster running TPC-H
queries with 30000 gigabyte of uncompressed data. Currently,
there are no official cluster results for more than 10000
gigabyte of data, where we achieve up to one to two orders of
magnitudes better performance than the current record holder.
%0 Conference Paper
%1 Weidner2013-qo
%A Weidner, M
%A Dees, J
%A Sanders, P
%B Big Data, 2013 IEEE International Conference on
%D 2013
%K Approximation_algorithms Approximation_methods Benchmark_testing CPU_utilization Cluster Context Data_analysis Data_warehouses Distributed Distributed_computing Distributed_databases Efficient_query_execution Message_passing Message_systems Parallel_processing Query_processing SQL TPC-H_benchmark_queries To_Read analytical_SQL_queries data_mining fast_OLAP_query_execution full_single_node_parallelization inter-node_communication main_memory_column-stores parallel_database_cluster parallel_databases precompiled_query_plans query_processing storage_management uncompressed_data
%P 518--524
%T Fast OLAP query execution in main memory on large data in a
cluster
%X Main memory column-stores have proven to be efficient for
processing analytical queries. Still, there has been little work
in the context of clusters. Using only a single machine poses
several restrictions: Processing power and data volume are
bounded to the number of cores and main memory fitting on one
tightly coupled system. To enable the processing of larger data
sets, switching to a cluster becomes necessary. In this work, we
explore techniques for efficient execution of analytical SQL
queries on large amounts of data in a parallel database cluster
while making maximal use of the available hardware. This
includes precompiled query plans for efficient CPU utilization,
full parallelization on single nodes and across the cluster, and
efficient inter-node communication. We implement all features in
a prototype for running a subset of TPC-H benchmark queries. We
evaluate our implementation in a 128 node cluster running TPC-H
queries with 30000 gigabyte of uncompressed data. Currently,
there are no official cluster results for more than 10000
gigabyte of data, where we achieve up to one to two orders of
magnitudes better performance than the current record holder.
@inproceedings{Weidner2013-qo,
abstract = {Main memory column-stores have proven to be efficient for
processing analytical queries. Still, there has been little work
in the context of clusters. Using only a single machine poses
several restrictions: Processing power and data volume are
bounded to the number of cores and main memory fitting on one
tightly coupled system. To enable the processing of larger data
sets, switching to a cluster becomes necessary. In this work, we
explore techniques for efficient execution of analytical SQL
queries on large amounts of data in a parallel database cluster
while making maximal use of the available hardware. This
includes precompiled query plans for efficient CPU utilization,
full parallelization on single nodes and across the cluster, and
efficient inter-node communication. We implement all features in
a prototype for running a subset of TPC-H benchmark queries. We
evaluate our implementation in a 128 node cluster running TPC-H
queries with 30000 gigabyte of uncompressed data. Currently,
there are no official cluster results for more than 10000
gigabyte of data, where we achieve up to one to two orders of
magnitudes better performance than the current record holder.},
added-at = {2015-04-11T18:41:09.000+0200},
author = {Weidner, M and Dees, J and Sanders, P},
biburl = {https://www.bibsonomy.org/bibtex/29c7be72d0799a38c33d393543906381a/christophv},
booktitle = {Big Data, 2013 {IEEE} International Conference on},
interhash = {8ac2001f857b5ea4280c974755855b3c},
intrahash = {9c7be72d0799a38c33d393543906381a},
keywords = {Approximation_algorithms Approximation_methods Benchmark_testing CPU_utilization Cluster Context Data_analysis Data_warehouses Distributed Distributed_computing Distributed_databases Efficient_query_execution Message_passing Message_systems Parallel_processing Query_processing SQL TPC-H_benchmark_queries To_Read analytical_SQL_queries data_mining fast_OLAP_query_execution full_single_node_parallelization inter-node_communication main_memory_column-stores parallel_database_cluster parallel_databases precompiled_query_plans query_processing storage_management uncompressed_data},
month = oct,
pages = {518--524},
timestamp = {2015-04-11T18:41:09.000+0200},
title = {Fast {OLAP} query execution in main memory on large data in a
cluster},
year = 2013
}