See the Go SDK Developer README at the sdks/go root for getting started with working on the Go SDK. In particular, the SDK is currently opinionated about where the beam repo needs to be copied in order for Go to use your SDK changes rather than it's own cached copy.
Contribution Recommendations
If you're looking to contribute to the Apache Beam and wish to do so with the Go SDK, you're welcome to! This section links to pages with recommendations of what and how to contribute.
- Supporting Streaming in the Go SDK - Help make the Go SDK feature complete, and more efficient!
- BEAM-11077 - Help make the Python Portable runner easier to use with the Go SDK!
- BEAM-11076 - Help make the Go Direct Runner useful, and portable!
- Current Go SDK Roadmap - What's being worked on at present.
- Add the Go SDK to the Beam Programming Guide!
- Make the beam package GoDoc more useful!
- File and add additional JIRAs!
Development Recommendations
- Follow the instructions in Go SDK Developer README at the sdks/go root
- Use the Python Portable Runner for end to end testing and validation of pipelines
- Follow the instructions in Python Tips to have a working python SDK environment.
- Ensure you're in the python virtual env.
- From the sdks/python directory, start a stand alone Python Portable Runner Job Service
python -m apache_beam.runners.portability.local_job_service_main --port=8099
- Submit jobs with the flags :
--runner=universal --endpoint=localhost:8099 --environment_type=LOOPBACK
- If this seems like too many steps, consider making the Python Portable runner more easily accessible to the Go SDK – BEAM-11077
- Consider alternatively using a local Spark or Flink instance to test pipelines. They can be spun up via beam repo gradle commands:
- Flink
./gradlew :runners:flink:1.16:job-server:runShadow
- Spark
./gradlew :runners:spark:3:job-server:runShadow
- Flink
- Do not use the "Go Direct" runner. It's not much more than a toy suitable for limited basic batch testing. It's a separate task to make it more generally useful – BEAM-11076
Testing
ValidatesRunner Tests
ValidatesRunner tests are basic integration tests of SDK functionality, such as primitive transforms, that can be run on assorted runners. These tests are present in beam/sdks/go/test/integration
.
Adding Tests
The documentation for package integration
contains information on adding new tests. In general, these are the main differences from regular unit tests:
- All integration tests should be in a subpackage, not in the root
integration
package. - Integration tests should be executing pipelines with
ptest.Run()
. Tests that do not execute pipelines likely do not belong as integration tests.- If this is a new package, it's required that
TestMain(m *testing.M)
is defined and callsptest.Main(m)
.
- If this is a new package, it's required that
The integration package defines a way to enable test filtering for your integration test. Call
integration.CheckFilters(t)
at the start of the test. integration.go contains the various runner specific filters, to filter out tests per runner, if they don't support the given feature under test. All integration tests should have test filtering enabled.- The test filter will fail the test in the event
TestMain
doesn't callptest.Main
.
- The test filter will fail the test in the event
Running Tests
- Manually: Tests can be run manually with the usual go test command by including any necessary flags. This approach is useful for running specific tests, but it requires any necessary services, such as job servers and cross-language expansion services, to be available with endpoints to use. For example:
go test -v ./sdks/go/test/integration/... --runner=portable --endpoint=localhost:8099
- Script: The script
beam/sdks/go/test/run_validatesrunner_tests.sh
can be run to provide some automation, and is more convenient for running a runner with more complex configuration, such as Dataflow. - Gradle: There are gradle tasks to build required services and run the script in
beam/sdks/go/test/build.gradle
. This approach is recommended for a convenient way of running the full integration test suite on a single runner. For example:./gradlew :clean :sdks:go:test:ulrValidatesRunner
Note on Cross-Language
Container issues have been known to occur when running cross-language pipelines with outdated vendored code in other languages. If you encounter container issues while running cross-language tests, perform the following steps to rebuild the docker container from up-to-date code:
- Delete your local containers for the language in question, like so:
docker image rm <image_id>
./gradlew :clean
- Rerun the gradle command to rebuild the containers and run the integration tests.
Cross-Language Quickstart
Go Cross-Language IO Quickstart Slides
The above slides are from a talk given to beginner developers on the Beam Go SDK, and may be a good starting point for future Beam developers.