Dataproc

Google Cloud Next ’20: OnAir | Learn what's new in Dataproc and OSS data processing, now available on demand. Watch session DA303 today.

Jump to

Dataproc makes open source data and analytics processing fast, easy, and more secure in the cloud. Try Dataproc free ● Created with Sketch. Spin up an autoscaling cluster in 90 seconds on custom machines ● Created with Sketch. Build fully managed Apache Spark, Apache Hadoop, Presto, and other OSS clusters ● Created with Sketch. Only pay for the resources you use and lower the total cost of ownership of OSS ● Created with Sketch. Encryption and unified security built into every cluster ● Created with Sketch. Accelerate data science with purpose-built clusters Dataproc product tutorial video

03:07 VIDEO

See Dataproc in action

Build custom OSS clusters on custom machines faster

Whether you need extra memory for Presto or GPUs for Apache Spark machine learning, Dataproc can help accelerate your data and analytics processing by spinning up a purpose-built cluster in 90 seconds.

Easy and affordable cluster management

With autoscaling, idle cluster deletion, per-second pricing, and more, Dataproc can help reduce the total cost of ownership of OSS so you can focus your time and resources elsewhere.

Security built in by default

Encryption by default helps ensure no piece of data is unprotected. With JobsAPI and Component Gateway, you can define permissions for Cloud IAM clusters, without having to set up networking or gateway nodes.

Key features

Automated cluster management

Managed deployment, logging, and monitoring let you focus on your data, not on your cluster. Dataproc clusters are stable, scalable, and speedy.

Containerize OSS jobs

When you build your OSS jobs (e.g., Apache Spark) on Dataproc, you can quickly containerize them with Kubernetes and deploy them anywhere a GKE cluster lives.

Enterprise security

When you create a Dataproc cluster, you can enable Hadoop Secure Mode via Kerberos by adding a Security Configuration. Additionally, some of the most commonly used Google Cloud-specific security features used with Dataproc include default at-rest encryption, OS Login, VPC Service Controls, and Customer Managed Encryption Keys (CMEK). View all features Modernize your Data Lake demo video

3:39 VIDEO

See how Dataproc & Cloud Storage can help accelerate loan processing: Demo

Customers

How Vodafone Built a Data Platform on Google Cloud (Next ’19 UK)

47:17

Vodafone Group moves 600 on-premises Apache Hadoop servers to the cloud.

Story highlights

Migrated on-premises Apache Hadoop to Google Cloud 226 models running in production Two months to roll out Google Cloud to first country

Industry

Telecommunications

49:57 Twitter moved from on-premises Hadoop to Google Cloud to more cost-effectively store and query tweets, users, impressions, and more.

50:51 Pandora migrated 7 PB+ of data from their on-premises Hadoop data lake to Google Cloud to unlock processing scale and help lower costs.

00:00 Spinning up and down Dataproc clusters helped METRO reduce infrastructure costs by 30% to 50%. See all customers

What's new

Sign up for Google Cloud newsletters to receive product updates, event information, special offers, and more. Jupyter logo

Blog post New GA Dataproc features extend data science and ML capabilities Learn more Delta lake logo

Blog post Getting started with Iceberg and Delta Lake table formats on Dataproc Read the blog Kubernetes logo

Blog post Modernize Apache Spark with Dataproc on Kubernetes Learn more

Documentation

APIs & Libraries

Dataproc initialization actions

Add other OSS projects to your Dataproc clusters with pre-built initialization actions. Learn more APIs & Libraries

Open source connectors

Libraries and tools for Apache Hadoop interoperability. Learn more

Not seeing what you’re looking for?

View all product documentation

Explore more docs

Quickstarts Get a quick intro to using this product. How-to guides Learn to complete specific tasks with this product. Tutorials Browse walkthroughs of common uses and scenarios for this product. APIs & references View APIs, references, and other resources for this product. Release notes Read about the latest releases for Dataproc

Use cases

Use case Move your Hadoop and Spark clusters to the cloud Enterprises are migrating their existing on-premises Apache Hadoop and Spark clusters over to Dataproc to manage costs and unlock the power of elastic scale. With Dataproc, enterprises get a fully managed, purpose-built cluster that can autoscale to support any data or analytics processing job.

Best practice

Apache Spark migration guide

Don’t rewrite your Spark code in Google Cloud. Learn more Best practice

Migrate HDFS data to Google Cloud

Learn when and how you should migrate your on-premises HDFS data to Google Cloud Storage. Learn more Best practice

Moving security controls from on-premises to Dataproc

Migrate existing security controls to Dataproc to help achieve enterprise and industry compliance. Learn more Use case Data science on Dataproc Create your ideal data science environment by spinning up a purpose-built Dataproc cluster. Integrate open source software like Apache Spark, NVIDIA RAPIDS, and Jupyter notebooks with Google Cloud AI services and GPUs to help accelerate your machine learning and AI development.

Tutorial

Use Dataproc and Apache Spark ML for machine learning

Integrate Dataproc with other Google Cloud services to build an end-to-end data science experience. Learn more Tutorial

PySpark for natural language processing on Dataproc

Open source libraries are key in order to accelerate machine learning development. Learn how to run NLP algorithms on Dataproc. Learn more Tutorial

Dataproc meets TensorFlow on YARN

Learn how to orchestrate distributed TensorFlow with TonY. Learn more View all technical guides

All features

Resizable clusters	Create and scale clusters quickly with various virtual machine types, disk sizes, number of nodes, and networking options.
Autoscaling clusters	Dataproc autoscaling provides a mechanism for automating cluster resource management and enables automatic addition and subtraction of cluster workers (nodes).
Cloud integrated	Built-in integration with Cloud Storage, BigQuery, Cloud Bigtable, Cloud Logging, Cloud Monitoring, and AI Hub, giving you a more complete and robust data platform.
Versioning	Image versioning allows you to switch between different versions of Apache Spark, Apache Hadoop, and other tools.
Highly available	Run clusters in high availability mode with multiple master nodes and set jobs to restart on failure to help ensure your clusters and jobs are highly available.
Cluster scheduled deletion	To help avoid incurring charges for an inactive cluster, you can use Dataproc's scheduled deletion, which provides options to delete a cluster after a specified cluster idle period, at a specified future time, or after a specified time period.
Automatic or manual configuration	Dataproc automatically configures hardware and software but also gives you manual control.
Developer tools	Multiple ways to manage a cluster, including an easy-to-use web UI, the Cloud SDK, RESTful APIs, and SSH access.
Initialization actions	Run initialization actions to install or customize the settings and libraries you need when your cluster is created.
Optional components	Use optional components to install and configure additional components on the cluster. Optional components are integrated with Dataproc components and offer fully configured environments for Zeppelin, Druid, Presto, and other open source software components related to the Apache Hadoop and Apache Spark ecosystem.
Custom images	Dataproc clusters can be provisioned with a custom image that includes your pre-installed Linux operating system packages.
Flexible virtual machines	Clusters can use custom machine types and preemptible virtual machines to make them the perfect size for your needs.
Component Gateway and notebook access	Dataproc Component Gateway enables secure, one-click access to Dataproc default and optional component web interfaces running on the cluster.
Workflow templates	Dataproc workflow templates provide a flexible and easy-to-use mechanism for managing and executing workflows. A workflow template is a reusable workflow configuration that defines a graph of jobs with information on where to run those jobs.

Pricing

Dataproc pricing is based on the number of vCPU and the duration of time that they run. While pricing shows hourly rate, we charge down to the second so you only pay for what you use. Please see pricing page for details View pricing details

Partners

Dataproc integrates with key partners to complement your existing investments and skill sets. Collibra logo

Take the next step

Start building on Google Cloud with $300 in free credits and 20+ always free products. Get started for free Need help getting started? Contact sales Work with a trusted partner Find a partner Continue browsing See all products ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Sign up for the Google Cloud newsletter

Aug	SEP	Oct
	17
2019	2020	2021