Skip to content

Commit

Permalink
[SPARK-44681] Fix issues when writing Go application code using Spark…
Browse files Browse the repository at this point in the history
… Connect Go client library

### What changes were proposed in this pull request?

When trying to write Go application code using Spark Connect Go client and put the Go application code in users' own repo (e.g. https://github.com/user-foo/my-go-application), the Go application code cannot resolve Spark Connect Go client from `https://github.com/apache/spark-connect-go` correctly, due to two issues:

1. The name `github.com/apache/spark-connect-go/v_3_4` cannot resolve correctly to find the module, it complains not finding `go.mod` file under `github.com/apache/spark-connect-go/v_3_4`. After change the name to `github.com/apache/spark-connect-go/v34`, it is good.

2. The Go application code needs generated Go protobuf code as well. Thus need to commit the generated Go protobuf code into `github.com/apache/spark-connect-go` repo.

### Why are the changes needed?

See above.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Tested by run Go application code to use Spark Connect Go client.

Closes #14 from hiboyang/bo-dev-05.

Authored-by: hiboyang <14280154+hiboyang@users.noreply.github.com>
Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
  • Loading branch information
hiboyang authored and zhengruifeng committed Aug 18, 2023
1 parent b35574e commit f2c9478
Show file tree
Hide file tree
Showing 20 changed files with 25,668 additions and 12 deletions.
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@
#

# All generated files
internal/generated
internal/generated.out

# Ignore Coverage Files
Expand Down
2 changes: 1 addition & 1 deletion client/channel/channel_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ import (
"strings"
"testing"

"github.com/apache/spark-connect-go/v_3_4/client/channel"
"github.com/apache/spark-connect-go/v34/client/channel"
"github.com/stretchr/testify/assert"
)

Expand Down
2 changes: 1 addition & 1 deletion client/sql/dataframe.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import (
"github.com/apache/arrow/go/v12/arrow"
"github.com/apache/arrow/go/v12/arrow/array"
"github.com/apache/arrow/go/v12/arrow/ipc"
proto "github.com/apache/spark-connect-go/v_3_4/internal/generated"
proto "github.com/apache/spark-connect-go/v34/internal/generated"
"io"
)

Expand Down
2 changes: 1 addition & 1 deletion client/sql/dataframe_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import (
"github.com/apache/arrow/go/v12/arrow/float16"
"github.com/apache/arrow/go/v12/arrow/ipc"
"github.com/apache/arrow/go/v12/arrow/memory"
proto "github.com/apache/spark-connect-go/v_3_4/internal/generated"
proto "github.com/apache/spark-connect-go/v34/internal/generated"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"testing"
Expand Down
2 changes: 1 addition & 1 deletion client/sql/dataframereader.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
package sql

import proto "github.com/apache/spark-connect-go/v_3_4/internal/generated"
import proto "github.com/apache/spark-connect-go/v34/internal/generated"

// DataFrameReader supports reading data from storage and returning a data frame.
// TODO needs to implement other methods like Option(), Schema(), and also "strong typed"
Expand Down
2 changes: 1 addition & 1 deletion client/sql/dataframewriter.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ package sql

import (
"fmt"
proto "github.com/apache/spark-connect-go/v_3_4/internal/generated"
proto "github.com/apache/spark-connect-go/v34/internal/generated"
"strings"
)

Expand Down
2 changes: 1 addition & 1 deletion client/sql/dataframewriter_test.go
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
package sql

import (
proto "github.com/apache/spark-connect-go/v_3_4/internal/generated"
proto "github.com/apache/spark-connect-go/v34/internal/generated"
"github.com/stretchr/testify/assert"
"testing"
)
Expand Down
4 changes: 2 additions & 2 deletions client/sql/sparksession.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ import (
"errors"
"fmt"

"github.com/apache/spark-connect-go/v_3_4/client/channel"
proto "github.com/apache/spark-connect-go/v_3_4/internal/generated"
"github.com/apache/spark-connect-go/v34/client/channel"
proto "github.com/apache/spark-connect-go/v34/internal/generated"
"github.com/google/uuid"
"google.golang.org/grpc/metadata"
"io"
Expand Down
2 changes: 1 addition & 1 deletion cmd/spark-connect-example-raw-grpc-client/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ package main
import (
"context"
"flag"
proto "github.com/apache/spark-connect-go/v_3_4/internal/generated"
proto "github.com/apache/spark-connect-go/v34/internal/generated"
"github.com/google/uuid"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
Expand Down
2 changes: 1 addition & 1 deletion cmd/spark-connect-example-spark-session/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ import (
"flag"
"log"

"github.com/apache/spark-connect-go/v_3_4/client/sql"
"github.com/apache/spark-connect-go/v34/client/sql"
)

var (
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.

module github.com/apache/spark-connect-go/v_3_4
module github.com/apache/spark-connect-go/v34

go 1.19

Expand Down
Loading

0 comments on commit f2c9478

Please sign in to comment.