This document provides additional information regarding the Beam Transform service. For primary document regarding the Transform service, please see the Beam programming guide.

Transforms included in the Transform service

This section includes a list of Beam transforms currently included in the Transform service. Following information regarding the transforms are provided.

Transform - a short description about the transform.
SDK implemented - SDK where the transform is implemented in.
Unique ID - a unique ID that can be used to access the transform. There are three types of unique IDs supported by the transform service.
- schema-aware: a schema-aware transform ID (only available for transforms implemented using Java). We recommend using schema-aware transforms over direct URN based transforms whenever available.
- urn: a URN that is used to register the transform in an expansion service
- python-name: Fully qualified class name or a callable (only available for transforms implemented using Python).

Transform	SDK implemented	Unique ID type	Unique ID
JDBC I/O read	Java	schema-aware	beam:schematransform:org.apache.beam:jdbc_read:v1
JDBC I/O write	Java	schema-aware	beam:schematransform:org.apache.beam:jdbc_write:v1
Pub/Sub I/O read	Java	schema-aware	beam:schematransform:org.apache.beam:pubsub_read:v1
Pub/Sub I/O write	Java	schema-aware	beam:schematransform:org.apache.beam:pubsub_write:v1
Pub/Sub Lite I/O read	Java	schema-aware	beam:schematransform:org.apache.beam:pubsublite_read:v1
Pub/Sub Lite I/O write	Java	schema-aware	beam:schematransform:org.apache.beam:pubsublite_write:v1
Kafka I/O read	Java	schema-aware	beam:schematransform:org.apache.beam:kafka_read:v1
Kafka I/O write	Java	schema-aware	beam:schematransform:org.apache.beam:kafka_write:v1
BigQuery I/O read (Storage Read API)	Java	schema-aware	beam:schematransform:org.apache.beam:bigquery_storage_read:v1
BigQuery I/O read (BQ file export)	Java	schema-aware	beam:schematransform:org.apache.beam:bigquery_export_read:v1
BigQuery I/O write (Storage Write API)	Java	schema-aware	beam:schematransform:org.apache.beam:bigquery_storage_write:v1
BigQuery I/O write (BQ file load jobs)	Java	schema-aware	beam:schematransform:org.apache.beam:bigquery_fileloads_write:v1
BigTable I/O read	Java	schema-aware	beam:schematransform:org.apache.beam:bigtable_read:v1
BigTable I/O write	Java	schema-aware	beam:schematransform:org.apache.beam:bigtable_write:v1
Spanner I/O CDC read	Java	schema-aware	beam:schematransform:org.apache.beam:spanner_cdc_read:v1
Spanner I/O write	Java	schema-aware	beam:schematransform:org.apache.beam:spanner_write:v1
JDBC I/O read	Java	urn	beam:transform:org.apache.beam:schemaio_jdbc_read:v1
JDBC I/O write	Java	urn	beam:transform:org.apache.beam:schemaio_jdbc_write:v1
Pub/Sub I/O read	Java	urn	beam:transform:org.apache.beam:pubsub_read:v1
Pub/Sub I/O write	Java	urn	beam:transform:org.apache.beam:pubsub_write:v1
Avro I/O read	Java	urn	beam:transform:org.apache.beam:schemaio_avro_read:v1
Avro I/O write	Java	urn	beam:transform:org.apache.beam:schemaio_avro_write:v1
BigQuery I/O read	Java	urn	beam:transform:org.apache.beam:schemaio_bigquery_read:v1
BigQuery I/O write	Java	urn	beam:transform:org.apache.beam:schemaio_bigquery_write:v1
Datastore I/O read	Java	urn	beam:transform:org.apache.beam:schemaio_datastoreV1_read:v1
Datastore I/O write	Java	urn	beam:transform:org.apache.beam:schemaio_datastoreV1_write:v1
Kafka I/O read without metadata	Java	urn	beam:transform:org.apache.beam:kafka_read_without_metadata:v1
Kafka I/O read with metadata	Java	urn	beam:transform:org.apache.beam:kafka_read_with_metadata
Kafka I/O write	Java	urn	beam:transform:org.apache.beam:kafka_write:v1
Pub/Sub I/O read	Java	urn	beam:transform:org.apache.beam:schemaio_pubsub_read:v1
Pub/Sub I/O write	Java	urn	beam:transform:org.apache.beam:schemaio_pubsub_write:v1
Pub/Sub Lite read	Java	urn	beam:transform:org.apache.beam:pubsublite_read:v1
Pub/Sub Lite write	Java	urn	beam:transform:org.apache.beam:pubsublite_write:v1
Spanner insert	Java	urn	beam:transform:org.apache.beam:spanner_insert:v1
Spanner update	Java	urn	beam:transform:org.apache.beam:spanner_update:v1
Spanner replace	Java	urn	beam:transform:org.apache.beam:spanner_replace:v1
Spanner insert or update	Java	urn	beam:transform:org.apache.beam:spanner_insert_or_update
Spanner delete	Java	urn	beam:transform:org.apache.beam:spanner_delete:v1
Spanner I/O read	Java	urn	beam:transform:org.apache.beam:spanner_read:v1
RunInference	Python	python-name	apache_beam.ml.inference.base.RunInference.from_callable
Dataframe	Python	python-name	apache_beam.dataframe.transforms.DataframeTransform

Upgrade transforms without upgrading the pipeline

Transform service allows Beam pipeline authors to upgrade specific transforms within their pipelines to a newer Beam version without upgrading the full pipeline. Please see the Beam programming guide for more details regarding this feature and supported SDKs and Beam versions.

Here we provide an example that upgrades a Beam Java BigQuery transforms of a pipeline to a newer Beam version without upgrading the pipeline.

Setup the environment according to the Beam Java quickstart guide.
Install Docker if not already available in the system.
Checkout the Beam examples Maven archetype for the relevant Beam version.

export BEAM_VERSION=<Beam version> # Needs to be Beam 2.53.0 or later.

mvn archetype:generate \
    -DarchetypeGroupId=org.apache.beam \
    -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
    -DarchetypeVersion=$BEAM_VERSION \
    -DgroupId=org.example \
    -DartifactId=beam-transform-upgrade \
    -Dversion="0.1" \
    -Dpackage=org.apache.beam.examples \
    -DinteractiveMode=false

cd beam-transform-upgrade

Add the Beam examples Java dependency to the <dependencies> section of the Maven pom.xml file.

<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-examples-java</artifactId>
<version>${beam.version}</version>
</dependency>

Execute the pipeline using a portable Beam runner. Following example uses Dataflow Runner v2.

export GCP_PROJECT=<GCP project>
export GCP_BUCKET=<GCP bucket>
export GCP_REGION=<GCP region>
export OUTPUT_BIGQUERY_TABLE=<A BigQuery table to write the output to>
export TRANSFORM_BEAM_VERSION=<Beam version to upgrade transforms to> # Needs to be Beam 2.53.0 or later.

mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.cookbook.BigQueryTornadoes -Dexec.args="--runner=DataflowRunner --project=$GCP_PROJECT \
                 --region=$GCP_REGION \
                 --tempLocation=gs://$GCP_BUCKET/transform-upgrade/tmp \
                 --experiments=use_runner_v2 --output=$OUTPUT_BIGQUERY_TABLE \
                 --transformsToOverride=beam:transform:org.apache.beam:bigquery_read:v1,beam:transform:org.apache.beam:bigquery_write:v1 \
                 --transformServiceBeamVersion=$TRANSFORM_BEAM_VERSION" -Pdataflow-runner