Here's a cloud round up of all things Hava, GCP, Azure and AWS for the week ending Friday December 9th 2022.
This week at Hava saw the release of self-hosted v2.1.522 which delivers new features and updates in response to package security recommendations. Hava customers with self-hosted deployments should upgrade to take advantage of the additional features.
To stay in the loop, make sure you subscribe using the box on the right of this page.
Of course we'd love to keep in touch at the usual places. Come and say hello on:
Now, customers can start using AWS IoT TwinMaker without having to recreate AWS IoT SiteWise assets and asset models, and any updates will be automatically synced. There is no charge for entities synced from AWS IoT SiteWise. If synced assets and asset models are no longer needed, customers can simply delete the sync setup.
AWS customers can use both arithmetic operators (such as +, -, /, and *) and mathematical functions (such as Sum and Average) to easily create custom metrics based on existing CloudWatch metrics. Target Tracking, like other EC2 Auto Scaling policies, helps customers maintain high availability while reducing costs by auto scaling their environments to meet changing demand.
Specifically, Target Tracking works like a thermostat: it constantly changes the capacity of an Auto Scaling group to maintain the specified metric at a customer-defined target level. Today’s release makes it easier and cheaper to configure Target Tracking with custom metrics.
Target Tracking offers out-of-the-box support for the most common infrastructure metrics such as CPU Utilization. In some cases, customers want to scale on their own application-specific metrics, such as the number of request served, or on metrics published by other AWS services , such as AWS SQS. Until today, you would have to create custom CloudWatch metrics for Target Tracking to consume.
Now, if the custom metric is a simple function of other existing metrics, you can use CloudWatch Metric Math in the Target Tracking policy, instead of publishing (and paying for) a new custom CloudWatch metric. For example, to define a custom metric representing the SQS messages per instance, you could take the existing SQS metric for queue length (ApproximateNumberOfMessages) and simply divide it by the number of instances in the Target Tracking Policy using Metric Math to make it work with your Target Tracking policy.
VPC IPAM allows you to easily organize IP addresses based on your routing and security needs and set simple business rules to govern IP address assignments. Using IPAM, you can automate IP address assignment to VPCs, eliminating the need to use spreadsheet-based or homegrown IP address planning applications, which can be hard to maintain and time-consuming.
IPAM also automatically tracks critical IP address information, eliminating the need to manually track or do bookkeeping for IP addresses. IPAM automatically retains your IP address monitoring data (up to a maximum of three years) which you can use to do retrospective analysis and audits for your network security and routing policies.
With this region expansion, VPC IPAM is now available in the following AWS Regions: Africa (Cape Town), Asia Pacific (Hong-Kong), Asia Pacific (Mumbai), Asia Pacific (Osaka), Asia Pacific (Sydney), Asia Pacific (Tokyo), Asia Pacific (Seoul), Asia Pacific (Singapore), Canada (Central), Europe (Dublin), Europe (Frankfurt), Europe (London), Europe (Milan), Europe (Paris), Europe (Stockholm), Middle East (Bahrain), South America (Sao Paulo), US West (Northern California), US East (N. Virginia), US East (Ohio), and US West (Oregon), AWS GovCloud (US-East) and AWS GovCloud (US-West).
Amazon QuickSight now supports even larger SPICE datasets on the Enterprise Edition. Today, all new SPICE datasets can accommodate up to 1 billion rows (or 1TB) of data in the Enterprise Edition and 25 million rows (or 25GB) for Standard Edition. Before the launch, each SPICE dataset could hold up to 500 million rows and 500GB of data.
Customers who want to use QuickSight's rich visualizations to explore very large datasets had to rely on data engineers to manually orchestrate data between QuickSight and another data store, which made it challenging to access and analyze very large datasets quickly. With billion-row support in SPICE, it’s easier to connect to data stores and ingest data into SPICE.
Users now have greater autonomy to visually analyze large datasets directly in QuickSight, without coordinating with engineering teams to manually orchestrate data among services. See here for details.
Support for billion-row SPICE dataset is now available in Amazon QuickSight Enterprise Editions in all QuickSight regions - US East (N. Virginia and Ohio), US West (Oregon), Canada, Sao Paulo, Europe (Frankfurt, Ireland and London), Asia Pacific (Mumbai, Seoul, Singapore, Sydney and Tokyo), and AWS GovCloud (US-West).
Amazon SageMaker Data Wrangler reduces the time it takes to aggregate and prepare data for machine learning (ML) from weeks to minutes in Amazon SageMaker Studio. With SageMaker Data Wrangler, you can simplify the process of data preparation and feature engineering, and complete each step of the data preparation workflow, including data selection, cleansing, exploration, and visualization from a single visual interface.
Starting today, you can connect to Amazon EMR Presto as a big query engine to bring in very large dataset, and prepare data for ML in minutes in Data Wrangler visual interactive.
Analyzing, transforming, and preparing large amounts of data is a critical part and also the most time-consuming part of ML workflow. Data scientists and data engineers leverage Apache Spark, Apache Hive, and Presto running on Amazon EMR for large scale data preparation.
Starting this week, customers can now use a visual interface to discover and connect to existing EMR clusters running Presto endpoint from Data Wrangler. They can browse the database, tables and schema, author Presto queries to select, preview and create a dataset for ML.
They can then use Data Wrangler visual interface to analyze data using Data Quality and Insights report, and clean data and create features for ML using 300+ built-in transformations backed by Spark without the need to author Spark code.
They can automatically train and deploy ML models using integration with SageMaker Autopilot. Finally, they can scale to process very large datasets with distributed processing jobs, automate data preparation using built-in scheduling capability, and run data prep in production workflows for training or inference with SageMaker Pipeline.
AWS Config now supports configuration recorder as a configuration item. Configuration recorder must be enabled before AWS Config can detect changes to your resource configurations and capture these changes as configuration items. With this launch, you can now monitor configuration changes to the configuration recorder in your AWS account.
AWS Config enables you to track and assess the configuration of your cloud resources throughout their life cycle. AWS Config console or AWS Command Line Interface (AWS CLI) users can update tracked resources or enable, disable or delete the configuration recorder.
The configuration recorder must stay enabled to run compliance evaluations for tracked resources. With this release, AWS allows you to track configuration changes to the configuration recorder’s state, specifically if it is no longer enabled or is uninstalled. This update also enables you to get an up-to-date list of resources that are tracked through AWS Config and run compliance checks for actively tracked resources.
This functionality is now available to all AWS Config users as default at no additional charge.
Starting this week, customers can view AWS CloudTrail event logs of a change request using AWS Systems Manager Change Manager. The feature helps customers understand which resources were impacted by the change request which provides customers with more visibility into the change request process.
Change Manager already helps customers request, approve, implement, and report on operational changes to their application configuration and infrastructure on AWS and on-premises. With this launch, customers can now get the CloudTrail events associated with the change requests within the Change Manager console which offers them more visibility into their changes.
For example, if a change was made to restart an EC2 instance, customers can now see which instances were actually impacted as part of the execution and the APIs that were invoked.
To get started, choose Change Manager from the AWS Systems Manager console in the left navigation menu to create a change request. After the change request is completed, the associated CloudTrail events are automatically displayed under the change request.
Amazon RDS Proxy, a fully managed, highly available database proxy for Amazon Relational Database Service (RDS), now supports creating proxies in Amazon Aurora Global Database primary and secondary regions. An Aurora Global Database is a single database that spans multiple AWS regions, enabling low latency global reads and disaster recovery from region-wide outages.
With this week’s launch, you can use RDS Proxy to make your applications more scalable, more resilient to database failures, and more secure in both the primary and secondary Global Database regions.
RDS Proxy sits between your application and the database to pool and share established database connections, improving database efficiency and application scalability. You can use RDS proxy to scale your applications read/write workloads in the Global Database primary region as well as read-only workloads on Aurora Replicas in both the primary and secondary regions.
In case of a failure, RDS Proxy automatically connects to a standby database instance within a region while preserving connections from your application and reduces failover times in the Global Database primary region. RDS Proxy can also help evenly distribute read-only workloads on Aurora Replicas within a AWS Region by taking into account database connection established to each Aurora Replica.
With Amazon RDS Proxy, database credentials and access can be managed through AWS Secrets Manager and AWS Identity and Access Management (IAM), eliminating the need to embed database credentials in the application.
AWS Cost Management now offers a 1-click experience to refresh Savings Plans Recommendations, so you can generate new Savings Plans Recommendations at any time to reduce costs and accelerate your cloud optimization journey.
Savings Plans is a flexible pricing model offering lower prices compared to On-Demand pricing, in exchange for a specific usage commitment (measured in $/hour) for a one or three-year period.
Savings Plans Recommendations enable you to reduce the cost of Amazon EC2, AWS Fargate, AWS Lambda, and Amazon SageMaker services by up to 72% while maintaining application performance.
With 1-click refresh, you can update your Savings Plans Recommendations within minutes to include newly purchased and expired Savings Plans along with your recent usage.
Amazon Kinesis Data Firehose now supports streaming data delivery to Logz.io, enabling Logz.io users to ingest streaming metrics and logs without having to manage applications or write code.
Amazon Kinesis Data Firehose makes it easier to reliably load streaming data into data lakes, data stores, and analytics services. You can use it can capture, transform, and deliver streaming data to Amazon S3, Amazon Redshift, Amazon OpenSearch Service, generic HTTP endpoints, and service providers like Logz.io.
It is a fully managed service that automatically scales to match the throughput of your data and requires no ongoing administration. With Amazon Kinesis Data Firehose, you don't need to write delivery applications or manage resources.
This new capability makes it easier than ever to route your AWS Service metrics and logs data directly into Logz.io. The Logz.io Cloud Observability Platform delivers the ELK Stack and Grafana as a fully-managed service so engineers can use the open source monitoring tools they know on a single solution, without the hassle of maintaining them at scale.
On top of the managed open source, Logz.io provides additional advanced analytics capabilities to make the ELK Stack and Grafana faster, more integrated, and easier to use.
Visit the Amazon Kinesis Console to configure your data producers to send data to Amazon Kinesis Data Firehose and specify Logz as the destination. Once set, Amazon Kinesis Data Firehose takes care of reliable, scalable delivery of your streaming data to Logz.
AWS are pleased to announce that as of this week, their customers will see additional details in AWS Cost Anomaly Detection’s console, alerting emails, and SNS topics posted to Slack and Chime. AWS Cost Anomaly Detection is a cost management service that leverages advanced machine learning to identify anomalous spend and root causes, so customers can quickly take action to avoid runaway spend and bill shocks.
With this launch, customer can spend less effort trying to understand what account and monitor is tied to a cost anomaly, which in turn helps them take necessary actions more quickly.
Customers will see account name, monitor name, and monitor type included in alert emails, the console, and notifications sent via SNS to Slack or Chime. In addition, start date, last detected date, and duration of an anomaly have been added to all email alerts.
This is both to help customers understand the anomaly details and also map it back to information already found in the cost anomaly details page.
Amazon EC2 customers can now use Recycle Bin for Amazon Machine Images (AMIs) in the Asia Pacific (Hyderabad) region to recover from accidental deletions to meet their business continuity needs. With Recycle Bin, you can specify a retention time period and recover a deregistered AMI if needed, before the expiration of the retention period. A recovered AMI would retain its attributes such as tags, permissions, and encryption status, which it had prior to deletion, and can be used immediately for launches.
AMIs that are not recovered from the Recycle Bin are permanently deleted upon expiration of the retention time.
You can enable Recycle Bin for all AMIs in your account by creating one or more retention rules. You can also use tags in retention rules to specify which subset of AMIs should move to the Recycle Bin upon deletion. Additionally, you can choose to lock your retention rules to prevent them from being unintentionally modified or deleted.
Amazon SageMaker Feature Store now supports the ability to create feature groups in the offline store in Apache Iceberg table format. The offline store contains historical ML features, organized into logical feature groups, and is used for model training and batch inference.
Apache Iceberg is an open table format for very large analytic datasets such as the offline store. It manages large collections of files as tables and supports modern analytical data lake operations optimized for usage on Amazon S3.
Ingesting data, especially when streaming, can result in a large number of small files which can negatively impact query performance due the higher number of file operations required. With Iceberg you can compact the small data files into fewer large files in the partition, resulting in significantly faster queries.
This compaction operation is concurrent and does not affect ongoing read and write operations on the feature group. If you chose the Iceberg option when creating new feature groups, SageMaker Feature Store will create the Iceberg tables using Parquet file format, and register the tables with the AWS Glue Data Catalog.
With the general availability of Amazon Data Lifecycle Manager in the AWS Middle East (UAE) Region, customers in that region can now automate the creation, sharing, copying and retention of Amazon EBS Snapshots and EBS-backed AMIs via policies. Data Lifecycle Manager eliminates the need for complicated custom scripts to manage your EBS resources, saving you time and money.
You can create policies that automatically create snapshots from EBS volumes and multi-volume crash-consistent snapshots of EBS Volumes attached to EC2 Instances. You can also configure your policies to copy tags from EC2 Instances and EBS Volumes to the snapshots that are created, as well as automatically copy your snapshots to another region or account for disaster recovery.
With EBS-backed AMI policies, you can set Data Lifecycle Manager to automatically, deprecate, and deregister AMIs and then delete the underlying snapshots, ensuring you do not pay for AMI snapshots that are no longer required.
Amazon Redshift Serverless, which allows you to run and scale analytics without having to provision and manage data warehouse clusters, is now generally available in additional AWS regions Asia Pacific (Mumbai) and Canada (Central).
With Amazon Redshift Serverless, all users including data analysts, developers, and data scientists, can use Amazon Redshift to get insights from data in seconds. Amazon Redshift Serverless automatically provisions and intelligently scales data warehouse capacity to deliver high performance for all your analytics.
You only pay for the compute used for the duration of the workloads on a per-second basis. You can benefit from this simplicity without making any changes to your existing analytics and business intelligence applications.
With a few clicks in the AWS Management Console, you can get started with querying data using the Query Editor V2 or your tool of choice with Amazon Redshift Serverless. There is no need to choose node types, node count, workload management, scaling, and other manual configurations.
You can take advantage of preloaded sample datasets along with sample queries to kick-start analytics immediately. You can create databases, schemas, and tables, and load your own data from Amazon S3, access data using Amazon Redshift data shares, or restore an existing Amazon Redshift provisioned cluster snapshot.
With Amazon Redshift Serverless, you can directly query data in open formats, such as Apache Parquet, in Amazon S3 data lakes and data in your operational databases, such as Amazon Aurora and Amazon Relational Database Service (Amazon RDS). Amazon Redshift Serverless provides unified billing for queries on any of these data sources, helping you efficiently monitor and manage costs.
Contact Lens for Amazon Connect now provides APIs that enable you to programmatically manage rules for Contact Lens’ conversational analytics and third party applications (e.g., Salesforce etc.). Rules allow you to automatically categorize contacts, send notifications about customer escalations, and assign tasks when a new case is created in your third party application.
Using the Rules APIs, you can now create new rules, search for specific rules, and update or delete an existing rule without requiring to use the Connect UI. For example, you can now use the CreateRule API to configure a new rule that can alert supervisors in real-time when customer sentiment turns negative on a call with an agent.
Additionally, to simplify and automate the setup of these APIs, Rules APIs also provides support for AWS CloudFormation.
Starting this week, Amazon EC2 Is4gen and Im4gn instances, the latest generation storage-optimized instances, are available in Europe (Paris) region. Based on the AWS Nitro System, Im4gn and Is4gen instances are powered by AWS Graviton2 processors and are built using AWS Nitro SSDs which reduce both latency and latency variability compared to the third generation of EC2 storage optimized instances.
Im4gn instances are optimized for applications requiring compute performance such as MySQL, NoSQL databases, and file systems. Is4gen instances provide dense SSD storage per vCPU for applications requiring high random I/O access to large amounts of local SSD data, such as stream processing and monitoring, real-time databases, and log analytics.
These instances can utilize AWS Elastic Fabric Adapter (EFA) and take advantage of Amazon’s Virtual Private Cloud (VPC) for security and reliability.
This week, AWS CloudFormation Hooks launched wildcard resource type for Hook configurations enabling customers to match multiple resources types. Customers can use wildcards to define resources targets for building flexible Hooks.
Such Hooks can activate for resource types which are not explicitly known when the Hook was created. For example, customers can use AWS::ECR::* wildcard to define a Hook that triggers for all resource types under Amazon ECR.
With this launch, customers can select a list of resource targets using “*” wildcard character, and match for prefix, infix, and suffix. For example, customers can select all resource types starting with AWS::S3 such as AWS::S3::AccessPoint, AWS::S3Outposts::Bucket, etc. with wildcard AWS::S3*.
This allows customer to automate resource validation to alert and/or prevent the provisioning operation of non-compliant resources.
This week, AWS were excited to announce that Amazon Transcribe Custom Language Models (CLM) now support German and Japanese languages in both batch and streaming mode. Amazon Transcribe is an automatic speech recognition (ASR) service that makes it easy for you to add speech-to-text capabilities to your applications.
CLM allows you to use pre-existing data to build a custom speech engine for your specific batch and streaming transcription use cases. No prior machine learning experience is required to create your CLM.
CLM uses text data that you already possess, such as website content, instruction manuals, and other assets that cover your domain’s unique lexicon and vocabulary. Upload your training dataset to create a CLM and run transcription jobs using your new CLM.
Amazon Transcribe CLM is meant for customers who operate in domains as diverse as law, finance, hospitality, insurance, and media. CLMs are designed to improve transcription accuracy for domain-specific speech. This includes any content outside of what you would hear in normal, everyday conversations.
For example, if you're transcribing the proceedings from a scientific conference, a standard transcription is unlikely to recognize many of the scientific terms used by presenters. Using Amazon Transcribe CLM, you can train a custom language model to recognize the specialized terms used in your discipline.
AWS Snow Family Large Data Migration Manager (LDMM) is now available in Asia Pacific (Hong Kong), Asia Pacific (Osaka), Europe (Milan), and Africa (Cape Town) Regions. Snow Large Data Migration Manager enables you to plan, track, and manage your large data migrations when using multiple Snowball Edge devices. You can now easily plan and monitor your jobs from a minimum of 500 Terabytes to Petabytes scale data migrations.
You can get started with the Large Data Migration Manager in three simple steps. First, go to the AWS Snow Family Management Console landing page and click on the ‘Create your large data migration plan' button. Second, create a new large data migration plan for your large migration project by providing the amount of data you need to migrate and location where Snow devices need to be shipped.
Third, once your large data migration plan is created, Large Data Migration Manager will show you options to either add existing jobs, or create new or clone old jobs and assign them to the plan. Once the jobs are added to your data migration plan, you can track the status of each Snow job and review the projected schedule for when to place Snow job orders.
This week, AWS IoT Device Defender launched a new audit check AWS IoT policy potentially misconfigured to identify certain potential misconfigurations in IoT policies. Security misconfigurations such as overly permissive policies can be a major cause of security incidents. With this new audit check in AWS IoT Device Defender, you can now more easily identify flaws, troubleshoot issues, and take the necessary corrective actions.
AWS IoT Device Defender helps in identifying IoT policies with permissive allow statements where devices could get access to unintended resources. It also inspects for use of MQTT wildcards in deny statements that could potentially be circumvented by devices when replacing wildcards with specific strings. This happens because MQTT wildcards don’t act as wildcard in IoT Core policies and are instead treated as literal strings.
This feature is available in all regions where AWS IoT Device Defender is available.
AWS CloudShell is now a System and Organization Controls (SOC) compliant service. You can now use AWS CloudShell for workloads that are subject to SOC compliance. AWS SOC reports are independent third-party examination reports that demonstrate how AWS achieves key compliance controls and objectives. The purpose of these reports is to help you and your auditors understand the AWS controls established to support operations and compliance.
AWS CloudShell is a browser-based shell that makes it easier to securely manage, explore, and interact with your AWS resources. CloudShell is pre-authenticated with your console credentials. Common development tools are pre-installed so no local installation or configuration is required.
With CloudShell you can run scripts with the AWS Command Line Interface (AWS CLI), define infrastructure with the AWS Cloud Development Kit (AWS CDK), experiment with AWS service APIs using the AWS SDKs, or use a range of other tools to increase your productivity.
Amazon Neptune Workbench is now available in Africa (Cape Town), Asia Pacific (Hong Kong), and Middle East (Bahrain) regions. Starting this week, you can use the Neptune Workbench to create Neptune notebook instances in these regions for visualizing graph data in Neptune, accessing tutorials, and running code samples using an interactive coding environment.
Amazon Neptune is a fast, reliable, and fully managed service that helps you build and run applications that work with highly connected datasets, such as knowledge graphs, fraud graphs, identity graphs, and security graphs. You can use the Neptune Workbench to visualize results of openCypher, Gremlin, and SPARQL queries, making it simple to get started with graph applications.
You can also use the Neptune Workbench magics to invoke the bulk loader, run query plans, and interactively build and train machine learning models using Neptune ML. Neptune notebooks are launched using the latest release of the open source GitHub project, graph-notebook.
To get started, create a new Neptune notebook in the AWS Management Console and check out the sample application tutorials packaged with every new notebook instance, such as “Building A Security Graph Application on Amazon Neptune”. For more details on visualization features in the Neptune Workbench, refer to the Amazon Neptune User Guide.
Refer to the Neptune pricing page for Neptune Workbench pricing and availability. Neptune notebooks are hosted by and billed as Amazon SageMaker Notebook Instances. Customers are charged for the notebook instance while the instance is in Ready state.
Amazon FSx for NetApp ONTAP, a service that provides fully managed shared storage built on NetApp’s popular ONTAP file system, today announced four new features that make it even easier to configure and manage your file systems.
You can now more easily assign a snapshot policy to your FSx for ONTAP volumes. Snapshots are browsable read-only images of a volume at a given point in time. Each FSx for ONTAP volume has a snapshot policy, which creates volume snapshots on a predefined schedule. Until today, you could only configure your volumes’ snapshot policies using the ONTAP CLI and REST API. Now, you can also assign a snapshot policy to new or existing volumes using the AWS Management Console and the Amazon FSx CLI and API, making it easier to configure when snapshots are automatically created for your volumes.
You can now more easily create FSx for ONTAP data protection (DP) volumes. DP volumes are used as the destination for NetApp SnapMirror, an ONTAP feature that enables you to efficiently replicate data to the same or different ONTAP file system. Until today, you could only create DP volumes using the ONTAP CLI and REST API. Now, you can also create DP volumes using the AWS Management Console and the Amazon FSx CLI and API, making it easier to migrate and protect your data with SnapMirror.
You can now configure FSx for ONTAP volumes so that their tags are automatically copied to backups you create. This capability makes it easier to organize, secure, track, and audit your backups.
You can now add or remove VPC route tables for your existing FSx for ONTAP Multi-AZ file systems. With Multi-AZ file systems, the endpoints you use to access and manage your data are created in VPC route tables you associate with your file system. Now, you can add or remove route tables for your existing file systems, enabling you to update your file system network configuration as your network evolves.
AWS are excited to announce that Amazon Lex now supports Arabic, Cantonese, Norwegian, Swedish, Polish, and Finnish. Amazon Lex is a service for building conversational interfaces into any application using voice and text.
Amazon Lex provides deep learning powered automatic speech recognition (ASR) for converting speech to text, and natural language understanding (NLU) to recognize the intent of the text, to enable you to build applications with highly engaging user experiences and lifelike conversational interactions. With these new languages, you can build and expand your conversational experiences to better understand and engage your customer base.
Previously, you could use NoSQL Workbench to visualize and build scalable, high-performance data models. However, to test DynamoDB API operations on your tables locally, you had to separately download and setup DynamoDB Local. With this new update in NoSQL Workbench, you can have DynamoDB Local up and running using a single NoSQL Workbench installer.
With just one tool to get started with DynamoDB, you can save time from doing additional tooling discovery and focus on developing your application in a self-contained local environment even when you have no internet connection to your Amazon Web Services (AWS) account.
Prior to this, you had to navigate between the NoSQL Workbench Data Modeler and the NoSQL Workbench homepage to download sample data model templates. With this new feature in NoSQL Workbench, you can directly import one of the many examples from the DynamoDB sample data model template library in the NoSQL Workbench Data Modeler when designing and optimizing for your applications.
Sensitive data detection and processing in AWS Glue is now generally available in 18 additional AWS Regions: US West (Northern California), Africa (Cape Town), Asia Pacific (Hong Kong), Asia Pacific (Mumbai), Asia Pacific (Osaka), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Canada (Central), Europe (Frankfurt), Europe (Ireland), Europe (London), Europe (Milan), Europe (Paris), Europe (Stockholm), Middle East (Bahrain), and South America (Sao Paulo).
This feature uses pattern matching and machine learning to automatically detect Personal Identifiable Information (PII) and other sensitive data at both the column and cell levels during an AWS Glue job run. AWS Glue includes options to log the type of PII and its location as well as to take action on it.
Sensitive data detection in AWS Glue identifies a variety of PII and other sensitive data like credit card numbers. It helps customers take action, such as tracking it for audit purposes or redacting the sensitive information before writing records into a data lake.
AWS Glue Studio’s visual, no-code interface lets users include Sensitive Data Detection as a step in a data integration job. It lets customers choose the type of personal information to detect as well as specify follow-on actions including redaction and logging. Customers can also define their own custom detection patterns for their unique needs.
This feature is now available in a total of 21 AWS Regions: US East (N. Virginia), US East (Ohio), US West (Northern California), US West (Oregon), Africa (Cape Town), Asia Pacific (Hong Kong), Asia Pacific (Mumbai), Asia Pacific (Osaka), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Canada (Central), Europe (Frankfurt), Europe (Ireland), Europe (London), Europe (Milan), Europe (Paris), Europe (Stockholm), Middle East (Bahrain), and South America (Sao Paulo).
AWS were excited to announce the launch of VMware Cloud on AWS in the AWS Cape Town region. This launch marks the 22nd AWS region to support VMware Cloud on AWS and provides customers running VMware-based workloads a faster and more seamless path to migrate to the cloud.
VMware customers in Africa region of Cape Town now have the option to extend their on-premises VMware workloads to the AWS Cloud. VMware Cloud on AWS is the only jointly engineered solution designed with speed and security in mind.
Accelerate your business transformation goals with a managed service that combines compute, network and storage capabilities in a service that is maintained, supported and operated by the creators of the software, VMware and the leading public cloud provider, AWS. VMware Cloud on AWS allows you to leverage the same familiar VMware tools, skill sets, and governance across your on-premises and cloud environments while benefitting from AWS cloud scale, economics and sustainability.
This allows customers to extend their on-premises datacenters, migrate their workloads and modernize these workloads through leveraging the 200+ AWS services available. Customers can deploy the entire VMware stack, consisting of ESXi, vCenter NSX and vSAN along with additional storage options.
VMware Cloud on AWS is the preferred public service provider for vSphere workloads, built on a sustainable architecture, which is committed to reaching net zero carbon emissions by 2030. These capabilities allow our customers to leverage the scale of all AWS data centers, through the new realization of the service within Cape Town.
Amazon FSx, a fully managed service that makes it easy to launch and run feature-rich and highly-performant file systems, is now authorized for Department of Defense Cloud Computing Security Requirements Guide Impact Levels 4 and 5 (DoD SRG IL4 and IL5) in the AWS GovCloud (US) Regions.
This launch builds on Amazon FSx’s existing DoD SRG IL2 authorization in AWS US Regions, FedRAMP Moderate authorization in US East (N. Virginia), US East (Ohio), US West (N. California), and US West (Oregon), and FedRAMP High authorization in AWS GovCloud (US) Regions.
Amazon FSx provides four file systems to choose from: NetApp ONTAP, OpenZFS, Windows File Server, and Lustre. The Defense Information Systems Agency (DISA) is an agency of the US Department of Defense (DoD) that is responsible for developing and maintaining the DoD Cloud Computing Security Requirements Guide (DoD SRG).
The Federal Risk and Authorization Management Program (FedRAMP) is a US government-wide program that delivers a standard approach to the security assessment, authorization, and continuous monitoring for cloud products and services.
Amazon FSx for NetApp ONTAP is extending the NVMe read cache support that is already included with Multi-AZ file systems to Single-AZ file systems. With the read cache, you can drive up to 650,000 IOPS and 6 GB/s of read throughput when reading your frequently-accessed data.
Amazon FSx for NetApp ONTAP has always included an NVMe read cache for Multi-AZ file systems, which reduces read latencies by up to 58%, increases IOPS by up to 4x, and increases throughput by up to 50% compared to data accessed from SSD storage.
Starting today, new Single-AZ file systems provisioned with at least 2 GB/s of throughput capacity include an NVMe read cache that is up to 5.7 TB in size and provides up to 650,000 IOPS and up to 6 GB/s of throughput. Now, you can extend the performance benefits that the NVMe read cache offers on Multi-AZ file systems to Single-AZ file systems.
AWS Security Hub now integrates with AWS Control Tower, allowing you to pair AWS Security Hub detective controls with AWS Control Tower proactive or preventive controls and manage them together using AWS Control Tower.
AWS Security Hub controls are now mapped to related control objectives in the AWS Control Tower control library, providing you with a holistic view of the controls required to meet a specific control objective. This combination of over 160 detective controls from AWS Security Hub, with the AWS Control Tower built-in automations for multi-account environments, gives you a strong baseline of governance and off-the-shelf controls required to scale your business using new AWS workloads and services.
This combination of controls also helps you monitor whether your multi-account AWS environment is secure and managed in accordance with best practices, such as the AWS Foundational Security Best Practices standard.
To use AWS Security Hub controls within AWS Control Tower, navigate to AWS Control Tower’s control library. After selecting any control that originates from AWS Security Hub, you can enable it directly from AWS Control Tower.
AWS Control Tower will activate AWS Security Hub on your behalf, and a new Service-Managed Standard will be created within AWS Security Hub. The new standard, managed by AWS Control Tower, allows you to see which AWS Security Hub controls have been activated by AWS Control Tower, and their evaluations.
To get started, visit the AWS Control Tower product page.
Amazon SageMaker Studio is a fully integrated development environment (IDE) for machine learning. Studio comes with built-in integration with Amazon EMR so that data scientists can interactively prepare data at petabyte scale using frameworks such as Apache Spark right from Studio notebooks.
AWS are excited to announce that SageMaker Studio now supports applying fine-grained data access control with AWS Lake Formation when accessing data through Amazon EMR.
Until now, all jobs that you ran on the EMR cluster used the same IAM role- the cluster’s EC2 Instance Profile - to access data. Therefore, to run jobs that needed access to different data sources e.g. different S3 buckets, you had to configure the EC2 Instance Profile with policies that allowed access to the union of all such data sources.
Additionally, to enable groups of users with differential access to data, you had to create separate clusters, one for each group, resulting in operational overhead. Separately, jobs submitted to EMR from Studio notebooks were unable to apply fine-grained data access control with AWS LakeFormation.
Starting this week, when you connect to EMR clusters from SageMaker Studio notebooks, you can choose that IAM role (called runtime IAM Role) that you want to connect with. Apache Spark, Hive or Presto jobs created from Studio notebooks will access only the data and resources permitted by policies attached to the runtime role.
Also, when data is accessed from data lakes managed with AWS LakeFormation, you can enforce table and column-level access using policies attached to the runtime role. With this new capability, multiple SageMaker Studio users can connect to the same EMR cluster, each using a runtime role scoped with customized data access permissions.
User sessions are completely isolated from one another on the shared cluster. With this feature, customers can simplify provisioning of EMR clusters, thus reducing operational overhead and saving costs.
This feature is generally available in SageMaker Studio when connecting to Amazon EMR 6.9 in the following AWS Regions: US East (Ohio), US East (N. Virginia), US West (Oregon), Europe (Paris).
Starting this week, Amazon FSx for NetApp ONTAP provides automatic encryption of data in transit between Nitro-based compute instances and new FSx for ONTAP file systems.
With FSx for ONTAP, until today, you needed to configure Kerberos to encrypt data in transit over the SMB 3.0+ and NFSv3+ protocols. Starting today, FSx for ONTAP supports automatic, Nitro-based encryption of data in transit that doesn’t rely on Kerberos.
This new feature is designed to leverage Nitro-based offload capabilities to automatically encrypt in-transit traffic with no impact on network performance. With Nitro-based encryption, data is encrypted in transit when accessed directly from supported instance types in the same VPC (or peered VPC).
AlloyDB for PostgreSQL
AlloyDB cross-region replication replicates your primary cluster's data and resources. It makes the data and resources available in different regions, allowing disaster recovery in the event of an outage in the primary region.
Anthos Clusters on VMware
Anthos clusters on VMware 1.11.6-gke.18 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.11.6-gke.18 runs on Kubernetes 1.22.15-gke.3300.
The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.13, 1.12, and 1.11.
Anthos Config Management
Config Sync now ignores validating and applying any resource configuration that has the annotation
config.kubernetes.io/local-config with any value except for "false", instead of ignoring only when the value is "true". This is consistent with its behavior in kpt.
The following five metrics are removed because these metrics aren't needed for monitoring system performance or health:
For information on current metrics, see Monitor Config Sync.
The first edition of the Config Sync Service Level Indicators (SLIs) is published. You can set up alerts based on these SLIs, and get alerts if a threshold is hit. To learn more about the Config Sync SLIs, see Use Config Sync SLIs.
The constraint template library includes a new template:
K8sBlockAllIngress. For reference see Constraint template library.
The constraint template library includes a new template:
K8sBlockCreationWithDefaultServiceAccount. For reference see Constraint template library.
The constraint template library includes a new template:
K8sBlockObjectsOfType. For reference see Constraint template library.
The constraint template library includes a new template:
K8sEnforceCloudArmorBackendConfig. For reference see Constraint template library.
The constraint template library includes a new template:
K8sEnforceConfigManagement. For reference see Constraint template library.
The constraint template library includes a new template:
K8sRequireDaemonsets. For reference see Constraint template library.
The constraint template library includes a new template:
K8sRequireDefaultDenyEgressPolicy. For reference see Constraint template library.
The constraint template library includes a new template:
K8sRequireValidRangesForNetworks. For reference see Constraint template library.
The constraint template library includes a new template:
K8sRestrictRbacSubjects. For reference see Constraint template library.
The following enhancements are made to Config Sync metrics:
apply_duration_secondsmetrics to support longer durations.
last_sync_timestampmetric to prevent timeseries with empty commits.
apply_operationsmetric to track whether the operation is from the applier or the remediator.
errorclasslabel of the
For more details, see Monitor Config Sync.
Added resource tags to all Config Sync metrics to identify the source component. For more information, see Config Sync Metric Tags.
Fixed a known compatibility issue in Config Sync that was announced in Anthos Config Management 1.13.1 affecting Autopilot on GKE 1.23 and later. Config Sync is now compatible with Autopilot clusters on all supported GKE versions.
Various reliability and stability improvements to Config Sync.
GA release of Simplified Onboarding for Apigee X (Pay-as-you-go) in the Google Cloud console.
With this release, new Apigee customers using Pay-as-you-go pricing can quickly configure Apigee using a simplified onboarding flow accessible from the Google Cloud console.
The following changes were made to UDM Search. You can now do the following:
Cloud Asset Inventory
Preview: You can now query asset metadata via the Cloud Asset Inventory API or the Cloud console, without needing to export the data to a BigQuery table first. This feature is available as a preview for Security Command Center Premium customers.
The ability to configure deletion protection for a Cloud Bigtable table is now generally available (GA). This setting prevents deletion of the table, its column families, and the instance containing the table. See Modify deletion protection for instructions.
A new suite of client-side metrics for the Cloud Bigtable client for Java is generally available (GA) in versions 2.16.0 and later. To learn more about using the new monitoring metrics for performance optimization and troubleshooting, see the Client-side metrics overview.
Preview: Get estimated costs in the Google Cloud console
You can now estimate the cost of Compute Engine and Cloud Storage workloads in the Google Cloud console. The Cost Estimation tool provides estimates that also include any custom contract prices on your Cloud Billing account. These cost estimates can help you make more informed business decisions.
View expiring commitments and automatically renew resource-based commitments with the Committed use discount dashboard
In the Committed use discount dashboard, you can now see subscription expiration notifications for commitments that are expiring within the next 30 days. In the dashboard's auto renewal column, you can automatically renew your resource-based commitments.
For more information about viewing the Commitment dashboard, see Committed use discount overview.
Cloud Composer 1.20.1 and 2.1.1 release started on December 6, 2022. Get ready for upcoming changes and features as we roll out the new release to all regions. This release is in progress at the moment.
Scheduled snapshots provide additional support for running disaster recovery scenarios.
The following versions for Cloud Composer 1.20.1 and 2.1.1 are available:
(Available without upgrading) Allowed custom secondary IP range for pods is now narrower. You can now create Composer environments using IP ranges with
(Cloud Composer 2) The Composer Local Development CLI tool is now available to help streamline testing and developing using local Airflow environments with Composer 2.
(New environments only) Creating Cloud Composer 2 environments no longer depends on the
constraints/compute.requireOsLogin organization policy setting.
(Cloud Composer 2) Cloud Composer 2 environments now include the
composer-user-workloads namespace that you can use to run user-defined workloads.
Cloud Data Fusion
Cloud Data Fusion is available in the following region:
Features in 6.8.0:
Replication from Oracle to BigQuery using Datastream is generally available (GA).
Cloud Data Fusion supports BigQuery batch source pushdown.
Cloud Data Fusion supports AND triggers. You can create OR and AND triggers. Previously, all triggers were OR triggers.
In Cloud Data Fusion 6.8.0, Reference name is no longer mandatory for the following plugins:
For these plugins, their unique identifiers in lineage are generated based on their configuration properties. For example, project
ID+dataset+table is used as a unique identifier for BigQuery. This identifier can be seen on the lineage diagram. For more information, see Cloud Data Fusion Plugins.
Changes in 6.8.0:
For Replication jobs with an Oracle (by Datastream) source, ensured data consistency when multiple CDC events are generated with the same timestamp, by ordering events reliably.
For Oracle replication sources, added a purge policy for a Cloud Storage bucket created by the plugin where Datastream writes its output.
In the Oracle replication source, added the GCS Bucket Location property, where Datastream writes its output.
In the Oracle replication source, added the list of Datastream regions to the Region property. You no longer need to manually enter the Datastream region.
The Oracle replication source identifies each row by the Primary key of the table. Previously, the plugin identified each row by the
For Replication jobs, improved performance for Review Assessment.
Splitter Transformation based plugins have access to
In Wrangler, added the Average
arithmetic function, which calculates the average of the selected columns.
In Wrangler, Numeric functions support 3 or more columns.
In the Dataplex Sink plugin, added the Update Dataplex Metadata property, which adds support for updating metadata in Dataplex for newly generated data.
In the GCS Delete Action plugin, added support for bulk deletion of files and folders. You can use the (
*) wildcard character to represent any character.
Fixed in 6.8.0:
For custom Dataproc compute profiles, fixed the issue causing the wrong Cloud Storage bucket to be used to stage data. Cloud Data Fusion uses the bucket specified in the custom compute profile.
Fixed the issue in the BigQuery Replication Target plugin causing Replication jobs to fail when the BigQuery target table already existed. The new version of the plugin is used by default in new Replication jobs. To use the new plugin version in existing jobs, recreate the job.
Fixed an issue causing the Replication Assessment to get stuck when the Oracle (by Datastream) storage bucket property was empty, or had an invalid bucket name. Cloud Data Fusion returns a
400 error code during assessment when the property is empty or has an invalid bucket name.
Fixed an issue causing Replication jobs to fail when the source column name didn't comply with BigQuery table naming conventions. In 6.8.0, if a source column name doesn't comply with BigQuery naming conventions, Cloud Data Fusion replaces invalid characters with an underscore, prepends an underscore when the first character is a number, and truncates the name when it exceeds the maximum length.
In the File batch source, fixed an issue causing Get Schema to appear only when Format was set to
delimited. Get Schema appears for all formats.
Fixed an issue with the output schema when connecting a Splitter transformation with a Joiner transformation.
Fixed an issue causing imports in the Cloud Data Fusion UI to fail for pipelines exported through the Pipeline Microservices.
In the Oracle Batch Source, when the source data included fields with the
Numeric data type (undefined precision and scale), Cloud Data Fusion set the precision to
38 and the scale to
0. If any values in the field had scale other than
0, Cloud Data Fusion truncated the values, which could have resulted in data loss. If the scale for a field was overridden in the plugin output schema, the pipeline failed. For more information, see the CDAP 6.8.0 bug fixes.
In the Wrangler transformation, fixed the issue causing the pipeline to not fail when the Error Handling property was set to
Fail Pipeline. This happened when an error was returned, but no exception was thrown, and there were
0 records in the output. For example, this happened when one of the directives (such
as.parse-as-simple-date) failed because the input data wasn't in the correct format. This fix is not available by default. Instead, contact support if you need this fix for your projects.
In Wrangler, fixed the issue causing the Wrangler page to get stuck when a BigQuery table name contained non-alphanumeric characters, such as underscores. Wrangler imports BigQuery tables that follow BigQuery table naming conventions.
Upgrading the Cloud Data Fusion version for Replication jobs is broken. Upgrading Replication jobs to Cloud Data Fusion version 6.8.0 isn't recommended.
Cloud Load Balancing
Currently, health check probes for hybrid NEGs originate from Google's centralized health checking mechanism. If you cannot allow traffic that originates from the Google health check ranges to reach your hybrid endpoints and would prefer to have the health check probes originate from your own private IP addresses instead, speak to your Google account representative to get your project allowlisted for distributed Envoy health checks.
This feature is available in General availability for allowlisted projects only.
For public and private uptime checks, a new create flow is available in Public Preview. For private uptime checks, the Public Preview flow lets you create the Service Directory service and perform other prerequisite steps by using the Google Cloud console. For more information, see Create public uptime checks and Create private uptime checks.
Cloud Run support for a new second generation execution environment is now at generally availability (GA).
GCP identified an issue in how they calculate the Total Database Storage metric in multi-regional Spanner instances. This metric is used to calculate the charges for Spanner database storage.
Database storage is currently incorrectly reported lower than it actually is in multi-regional configurations, resulting in undercharging for database storage. We communicated a Service Announcement in October and started rolling out this change to pricing on December 1, 2022. Depending on your configuration, your Total Database Storage could increase by up to 25%.
For the majority of impacted customers, the impact on your total bill will be less than 0.5%. For those affected, you will notice an increase in database storage charges that reflect this corrected metric.
We waived the under-billed amount for all past billing cycles. Please note that this issue only affects multi-region configurations of Spanner. It does not affect regional configurations of Spanner. Additionally, the Total Backup Storage metric is not affected by this issue, and has always been reported correctly.
For more information, see Database storage prices.
New SQL syntax, RETURNING in the PostgreSQL dialect and THEN RETURN in Google Standard SQL, selects and returns data from rows that were just updated as part of a DML statement. This is especially useful for getting values from default or generated columns and can reduce latency over equivalent multi-statement transactions. The preview supports the Java, JDBC, Python, and Go Spanner clients as well as PostgreSQL drivers that connect through PGAdapter.
Generally available: You can merge your active hardware resource commitments into one larger commitment to track and manage them as a single entity. You can now also merge your commitments by using the Google Cloud Console. For more information, see Merging commitments.
GKE cluster versions have been updated.
Google Cloud VMware Engine
In order to support new features in the future, Google Cloud VMware Engine will convert the resource names for private clouds to a standardized format that is more consistent with Google Cloud. Specifically, this resource name translation will make minor changes to the names of resources in your project, such as:
Resource name translation is currently optional, but existing private clouds must perform a resource name translation in order to access the gcloud CLI or VMware Engine API. Resource name translation will be required after September 2023.
For more information on resource name translation, see Resource Name Translation
Security Command Center
Malicious URL Observed detector of Container Threat Detection, a built-in service of Security Command Center Premium, is now generally available.
The detector checks URLs observed in arguments passed by executables against known phishing and malware URLs to determine if they are malicious.
You can see the full details of the detector's findings only if you upgrade to the refreshed findings display in the Security Command Center dashboard.
For more information, see the following pages:
Sensitive Actions Service, a built-in service of Security Command Center Premium, is now generally available.
Sensitive Actions Service detects when actions are taken in your Google Cloud organization, folders, and projects that could be damaging to your business if they were to be taken by a malicious actor.
For more information, see Sensitive Actions Service overview.
For more information, see the Vertex AI Vision documentation.
The Pipeline Templates feature is now generally available (GA). The Your Templates tab is supported by Artifact Registry and allows you to publish and curate pipeline and component templatess. For documentation, refer to Create, upload, and use a pipeline template.
Microsoft Azure Releases And Updates
New VM sizes provide the best remote storage performance of any Azure VMs to date. You can now process more data with fewer vCPUs while potentially reducing software licensing costs.
This article describes the improvements for the latest version of Azure Site Recovery components.
Have you tried Hava automated diagrams for AWS, Azure, GCP and Kubernetes. Get back your precious time and sanity and rid yourself of manual drag and drop diagram builders forever.
Hava automatically generates accurate fully interactive cloud infrastructure and security diagrams when connected to your AWS, Azure, GCP accounts or stand alone K8s clusters. Once diagrams are created, they are kept up to date, hands free.
When changes are detected, new diagrams are auto-generated and the superseded documentation is moved to a version history. Older diagrams are also interactive, so can be opened and individual resources inspected interactively, just like the live diagrams.
Check out the 14 day free trial here (includes forever free tier):