Google Cloud has introduced some new tools for easier administration and management of its serverless modern data warehouse BigQuery. Optimizing BigQuery is as important as ever, as management requirements and monitoring continue to evolve while organizations utilize BigQuery’s analytics features. With new BigQuery Administrator Hub capabilities, customers can now better manage BigQuery at scale.
Within BigQuery Administrator Hub are Resource Charts and Slot Estimator features. These tools help administrators understand their BigQuery environments on a deeper level! Here’s how they work.
Real-Time Monitoring with Resource Charts
First, Resource Charts empower Administrators to monitor their slot usage, manage capacity based on historical consumption, troubleshoot job performance, and self-diagnose queries and take corrective action as needed.
Resource Charts also provide visibility into key metrics such as slot consumption, job performance, concurrency, bytes processed, and failed jobs. The new Errors chart allows users to investigate invalid errors and denied permissions. The Errors chart also provides users with filters to take a deeper look at any inconsistencies or issues!
They are built and rendered using INFORMATION_SCHEMA tables, enabling customers to understand data through these purpose-built dashboards or query the data directly to build their own dashboards and monitoring processes.
With Resource Charts providing real-time environment insights, users have the ability to see what is consuming slots and which workloads/queries are driving usage. Introduced earlier this year, Resource Charts are now generally available to BigQuery customers.
Manage Slot Capacity with Slot Estimator
Next, Slot Estimator is an interactive capacity management tool that helps administrators estimate and optimize their BigQuery capacity based on performance.
It provides capabilities for tasks such as:
- Viewing slot capacity and utilization data across all reservations in an organization.
- Identifying periods of peak utilization when the most slots are used.
- Viewing job latency percentiles to understand query performance.
- Modeling how increasing or reducing slots will affect performance.
For now, data is limited to the last seven days. Slot Estimator is available in preview for customers using Reservations.
Google Cloud’s BigQuery and You
Not sure where to begin with Google Cloud’s BigQuery? Pandera is here to assist! Even if you’re already familiar with modern data warehousing, our certified Google Cloud experts are able to assist organizations to reach their data-driven goals.
For a deeper understanding of Google Cloud Platform’s latest tools and how they can benefit your business, please get in touch with us today!