Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Project Rename and Postgres Changes #21

Merged
merged 3 commits into from
Jun 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
shift
synkit
.vscode
2 changes: 1 addition & 1 deletion LICENCE
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Shift License
Synkit License

Version 1.1, 30th June 2023

Expand Down
20 changes: 10 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Shift
# Synkit

Shift is a new data integration protocol and specification, written in Golang, that aims to provide an alternative to existing solutions such as Airbyte, Meltano, and Singer-io. It focuses on simplicity, performance, and extensibility to enable seamless data integration between various systems and applications.
Synkit is a new data integration protocol and specification, written in Golang, that aims to provide an alternative to existing solutions such as Airbyte, Meltano, and Singer-io. It focuses on simplicity, performance, and extensibility to enable seamless data integration between various systems and applications.

## Roadmap
- [ ] Postgres
Expand All @@ -11,33 +11,33 @@ Shift is a new data integration protocol and specification, written in Golang, t

## Getting Started

To start using Shift, follow these steps:
To start using Synkit, follow these steps:

1. **Run Shift**: Run Shift driver/adapter images directly with config, catalog, and state files.
2. **Explore the Documentation**: Refer to the documentation for detailed information on how to use Protocol Shift, including connector setup, data source configuration, and integration examples.
1. **Run Synkit**: Run Synkit driver/adapter images directly with config, catalog, and state files.
2. **Explore the Documentation**: Refer to the documentation for detailed information on how to use Protocol Synkit, including connector setup, data source configuration, and integration examples.

## Contributing

We welcome contributions to Protocol Shift! If you'd like to contribute, please follow these guidelines:
We welcome contributions to Protocol Synkit! If you'd like to contribute, please follow these guidelines:

1. Fork the repository and create a new branch for your feature or bug fix.

2. Commit your changes with descriptive commit messages.

3. Push your changes to your forked repository.

4. Submit a pull request to the main Shift repository, clearly describing the changes you've made and their purpose.
4. Submit a pull request to the main Synkit repository, clearly describing the changes you've made and their purpose.

5. Participate in the code review process, addressing any feedback or suggestions.

## License

Protocol Shift is open-source and released under a custom Licence. Please check
Protocol Synkit is open-source and released under a custom Licence. Please check

## Contact

For any questions, feedback, or support related to Shift, you can reach out to the development team via email at [piyushsingariya@gmail.com](mailto:piyushsingariya@gmail.com).
For any questions, feedback, or support related to Synkit, you can reach out to the development team via email at [piyushsingariya@gmail.com](mailto:piyushsingariya@gmail.com).

## Acknowledgments

We would like to acknowledge the contributions of all individuals and organizations that have helped make Shift possible. Their feedback, ideas, and suggestions have been invaluable in shaping and improving the protocol.
We would like to acknowledge the contributions of all individuals and organizations that have helped make Synkit possible. Their feedback, ideas, and suggestions have been invaluable in shaping and improving the protocol.
6 changes: 3 additions & 3 deletions build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,17 +17,17 @@ function build_and_run() {
fi
cd $path
go mod tidy
go build -ldflags="-w -s -X constants/constants.version=${GIT_VERSION} -X constants/constants.commitsha=${GIT_COMMITSHA} -X constants/constants.releasechannel=${RELEASE_CHANNEL}" -o shift main.go
go build -ldflags="-w -s -X constants/constants.version=${GIT_VERSION} -X constants/constants.commitsha=${GIT_COMMITSHA} -X constants/constants.releasechannel=${RELEASE_CHANNEL}" -o synkit main.go

echo "============================== Executing connector: $connector with args [$joined_arguments] =============================="
./shift $joined_arguments
./synkit $joined_arguments
}

if [ $# -gt 0 ]; then
argument="$1"

# Capture and join remaining arguments
shift
synkit
remaining_arguments=("$@")
joined_arguments=$(
IFS=' '
Expand Down
8 changes: 4 additions & 4 deletions connector.go
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
package shift
package synkit

import (
"fmt"
"os"

"github.com/piyushsingariya/shift/logger"
protocol "github.com/piyushsingariya/shift/protocol"
"github.com/piyushsingariya/shift/safego"
"github.com/piyushsingariya/synkit/logger"
protocol "github.com/piyushsingariya/synkit/protocol"
"github.com/piyushsingariya/synkit/safego"
"github.com/spf13/cobra"
)

Expand Down
6 changes: 3 additions & 3 deletions drivers/base/bulk_driver.go
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
package base

import (
"github.com/piyushsingariya/shift/protocol"
"github.com/piyushsingariya/shift/types"
"github.com/piyushsingariya/shift/utils"
"github.com/piyushsingariya/synkit/protocol"
"github.com/piyushsingariya/synkit/types"
"github.com/piyushsingariya/synkit/utils"
)

// Pass dest with all fields initialized to handle nil state case
Expand Down
6 changes: 3 additions & 3 deletions drivers/base/driver.go
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
package base

import (
"github.com/piyushsingariya/shift/protocol"
"github.com/piyushsingariya/shift/types"
"github.com/piyushsingariya/shift/typeutils"
"github.com/piyushsingariya/synkit/protocol"
"github.com/piyushsingariya/synkit/types"
"github.com/piyushsingariya/synkit/typeutils"
)

type Driver struct {
Expand Down
4 changes: 2 additions & 2 deletions drivers/base/utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ package base
import (
"time"

"github.com/piyushsingariya/shift/logger"
"github.com/piyushsingariya/shift/types"
"github.com/piyushsingariya/synkit/logger"
"github.com/piyushsingariya/synkit/types"
)

type basestream interface {
Expand Down
6 changes: 3 additions & 3 deletions drivers/google-sheets/go.mod
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
module github.com/piyushsingariya/shift/drivers/google-sheets
module github.com/piyushsingariya/synkit/drivers/google-sheets

go 1.22

Expand All @@ -7,7 +7,7 @@ toolchain go1.22.3
require (
github.com/goccy/go-json v0.10.2
github.com/mozillazg/go-unidecode v0.2.0
github.com/piyushsingariya/shift v0.0.0-00010101000000-000000000000
github.com/piyushsingariya/synkit v0.0.0-00010101000000-000000000000
golang.org/x/oauth2 v0.10.0
gopkg.in/Iwark/spreadsheet.v2 v2.0.0-20220412131121-41eea1483964
)
Expand Down Expand Up @@ -46,4 +46,4 @@ require (
sigs.k8s.io/yaml v1.3.0 // indirect
)

replace github.com/piyushsingariya/shift => ../../
replace github.com/piyushsingariya/synkit => ../../
10 changes: 5 additions & 5 deletions drivers/google-sheets/internal/google_sheets.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@ import (
"fmt"
"strings"

"github.com/piyushsingariya/shift/drivers/base"
"github.com/piyushsingariya/shift/logger"
"github.com/piyushsingariya/shift/protocol"
"github.com/piyushsingariya/shift/types"
"github.com/piyushsingariya/shift/utils"
"github.com/piyushsingariya/synkit/drivers/base"
"github.com/piyushsingariya/synkit/logger"
"github.com/piyushsingariya/synkit/protocol"
"github.com/piyushsingariya/synkit/types"
"github.com/piyushsingariya/synkit/utils"
"gopkg.in/Iwark/spreadsheet.v2"
)

Expand Down
2 changes: 1 addition & 1 deletion drivers/google-sheets/internal/models.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ package driver
import (
"fmt"

"github.com/piyushsingariya/shift/types"
"github.com/piyushsingariya/synkit/types"
)

type Config struct {
Expand Down
6 changes: 3 additions & 3 deletions drivers/google-sheets/internal/utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@ import (
"github.com/goccy/go-json"

unidecode "github.com/mozillazg/go-unidecode"
"github.com/piyushsingariya/shift/logger"
"github.com/piyushsingariya/shift/types"
"github.com/piyushsingariya/shift/utils"
"github.com/piyushsingariya/synkit/logger"
"github.com/piyushsingariya/synkit/types"
"github.com/piyushsingariya/synkit/utils"
"golang.org/x/oauth2"
"golang.org/x/oauth2/google"
"gopkg.in/Iwark/spreadsheet.v2"
Expand Down
6 changes: 3 additions & 3 deletions drivers/google-sheets/main.go
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
package main

import (
"github.com/piyushsingariya/shift"
driver "github.com/piyushsingariya/shift/drivers/google-sheets/internal"
"github.com/piyushsingariya/synkit"
driver "github.com/piyushsingariya/synkit/drivers/google-sheets/internal"
)

func main() {
driver := &driver.GoogleSheets{}
shift.RegisterDriver(driver)
synkit.RegisterDriver(driver)

Check failure on line 10 in drivers/google-sheets/main.go

View workflow job for this annotation

GitHub Actions / GoSec Security Scanner

cannot use driver (variable of type *"github.com/piyushsingariya/synkit/drivers/google-sheets/internal".GoogleSheets) as protocol.Driver value in argument to synkit.RegisterDriver: *"github.com/piyushsingariya/synkit/drivers/google-sheets/internal".GoogleSheets does not implement protocol.Driver (missing method Config)
}
Binary file added drivers/google-sheets/shift
Binary file not shown.
6 changes: 3 additions & 3 deletions drivers/hubspot/go.mod
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
module github.com/piyushsingariya/shift/drivers/hubspot
module github.com/piyushsingariya/synkit/drivers/hubspot

go 1.22

toolchain go1.22.3

require (
github.com/goccy/go-json v0.10.2
github.com/piyushsingariya/shift v0.0.0-20230727050722-6795340c7033
github.com/piyushsingariya/synkit v0.0.0-20230727050722-6795340c7033
golang.org/x/oauth2 v0.11.0
)

Expand Down Expand Up @@ -42,4 +42,4 @@ require (
sigs.k8s.io/yaml v1.3.0 // indirect
)

replace github.com/piyushsingariya/shift => ../../
replace github.com/piyushsingariya/synkit => ../../
10 changes: 5 additions & 5 deletions drivers/hubspot/internal/base_stream.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,11 @@

"github.com/goccy/go-json"

"github.com/piyushsingariya/shift/drivers/base"
"github.com/piyushsingariya/shift/logger"
"github.com/piyushsingariya/shift/types"
"github.com/piyushsingariya/shift/typeutils"
"github.com/piyushsingariya/shift/utils"
"github.com/piyushsingariya/synkit/drivers/base"
"github.com/piyushsingariya/synkit/logger"
"github.com/piyushsingariya/synkit/types"
"github.com/piyushsingariya/synkit/typeutils"
"github.com/piyushsingariya/synkit/utils"
)

type Stream struct {
Expand Down Expand Up @@ -158,7 +158,7 @@
}

func (s *Stream) ScopeIsGranted(grantedScopes []string) bool {
s.grantedScopes = types.Set(grantedScopes)

Check failure on line 161 in drivers/hubspot/internal/base_stream.go

View workflow job for this annotation

GitHub Actions / GoSec Security Scanner

cannot use generic type types.Set without instantiation
return utils.IsSubset(grantedScopes, s.scopes)
}

Expand Down
2 changes: 1 addition & 1 deletion drivers/hubspot/internal/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"fmt"
"time"

"github.com/piyushsingariya/shift/logger"
"github.com/piyushsingariya/synkit/logger"
)

type Config struct {
Expand Down
12 changes: 6 additions & 6 deletions drivers/hubspot/internal/crm_stream.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@ import (
"net/http"
"time"

"github.com/piyushsingariya/shift/drivers/base"
"github.com/piyushsingariya/shift/logger"
"github.com/piyushsingariya/shift/safego"
"github.com/piyushsingariya/shift/types"
"github.com/piyushsingariya/shift/typeutils"
"github.com/piyushsingariya/shift/utils"
"github.com/piyushsingariya/synkit/drivers/base"
"github.com/piyushsingariya/synkit/logger"
"github.com/piyushsingariya/synkit/safego"
"github.com/piyushsingariya/synkit/types"
"github.com/piyushsingariya/synkit/typeutils"
"github.com/piyushsingariya/synkit/utils"
)

type CRMSearchStream struct {
Expand Down
12 changes: 6 additions & 6 deletions drivers/hubspot/internal/hubspot.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@
"net/http"

"github.com/goccy/go-json"
"github.com/piyushsingariya/shift/drivers/base"
"github.com/piyushsingariya/synkit/drivers/base"

"github.com/piyushsingariya/shift/jsonschema"
"github.com/piyushsingariya/shift/jsonschema/schema"
"github.com/piyushsingariya/shift/protocol"
"github.com/piyushsingariya/shift/types"
"github.com/piyushsingariya/shift/utils"
"github.com/piyushsingariya/synkit/jsonschema"
"github.com/piyushsingariya/synkit/jsonschema/schema"
"github.com/piyushsingariya/synkit/protocol"
"github.com/piyushsingariya/synkit/types"
"github.com/piyushsingariya/synkit/utils"
)

type Hubspot struct {
Expand Down Expand Up @@ -65,7 +65,7 @@
WithPrimaryKey(hstream.PrimaryKey()...)
stream.WithCursorField(hstream.cursorField())

streams = append(streams, stream)

Check failure on line 68 in drivers/hubspot/internal/hubspot.go

View workflow job for this annotation

GitHub Actions / GoSec Security Scanner

cannot use stream (variable of type *"github.com/piyushsingariya/synkit/types".Stream) as protocol.Stream value in argument to append: *"github.com/piyushsingariya/synkit/types".Stream does not implement protocol.Stream (missing method BatchSize)
}

return streams, nil
Expand Down
8 changes: 4 additions & 4 deletions drivers/hubspot/internal/incremental_stream.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@
"net/http"
"time"

"github.com/piyushsingariya/shift/logger"
"github.com/piyushsingariya/shift/types"
"github.com/piyushsingariya/shift/typeutils"
"github.com/piyushsingariya/shift/utils"
"github.com/piyushsingariya/synkit/logger"
"github.com/piyushsingariya/synkit/types"
"github.com/piyushsingariya/synkit/typeutils"
"github.com/piyushsingariya/synkit/utils"
)

type IncrementalStream struct {
Expand Down Expand Up @@ -40,7 +40,7 @@
statePk: "timestamp",
}

s.availableSyncMode = append(s.availableSyncMode, types.Incremental)

Check failure on line 43 in drivers/hubspot/internal/incremental_stream.go

View workflow job for this annotation

GitHub Actions / GoSec Security Scanner

undefined: types.Incremental
return s
}

Expand Down
2 changes: 1 addition & 1 deletion drivers/hubspot/internal/interface.go
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
package driver

import (
"github.com/piyushsingariya/shift/types"
"github.com/piyushsingariya/synkit/types"
)

type HubspotStream interface {
Expand Down
6 changes: 3 additions & 3 deletions drivers/hubspot/internal/utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ import (
"reflect"
"strings"

"github.com/piyushsingariya/shift/logger"
"github.com/piyushsingariya/shift/types"
"github.com/piyushsingariya/shift/utils"
"github.com/piyushsingariya/synkit/logger"
"github.com/piyushsingariya/synkit/types"
"github.com/piyushsingariya/synkit/utils"
"golang.org/x/oauth2"
)

Expand Down
6 changes: 3 additions & 3 deletions drivers/hubspot/main.go
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
package main

import (
"github.com/piyushsingariya/shift"
driver "github.com/piyushsingariya/shift/drivers/hubspot/internal"
"github.com/piyushsingariya/synkit"
driver "github.com/piyushsingariya/synkit/drivers/hubspot/internal"
)

func main() {
driver := &driver.Hubspot{}
shift.RegisterDriver(driver)
synkit.RegisterDriver(driver)
}
Loading
Loading