I am trying to use AWS dynamodb for the KV store and I running into the following error

[kushal-otter]

Error:
panic: runtime error: invalid memory address or nil pointer dereference │
│ [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xc381bd] │
│ │
│ goroutine 1 [running]: │
│ github.com/aws/aws-sdk-go/aws/session.(*Session).resolveEndpoint(0x0?, {0x2a01650, 0x8}, {0xc000914f00?, 0x9}, {0x0, 0x0}, 0xc0002179e0) │
│ /__w/cortex/cortex/vendor/github.com/aws/aws-sdk-go/aws/session/session.go:907 +0x23d │
│ github.com/aws/aws-sdk-go/aws/session.(*Session).ClientConfig(, {, }, {, _, }) │
│ /__w/cortex/cortex/vendor/github.com/aws/aws-sdk-go/aws/session/session.go:871 +0xfa │
│ github.com/aws/aws-sdk-go/service/dynamodb.New({0x311cf60?, 0xc0007b8000?}, {0x0?, 0x445fb1?, 0x2668a00?}) │
│ /__w/cortex/cortex/vendor/github.com/aws/aws-sdk-go/service/dynamodb/service.go:54 +0x53 │
│ github.com/cortexproject/cortex/pkg/ring/kv/dynamodb.newDynamodbKV({{0xc000914f00, 0x9}, {0xc00091c858, 0x11}, 0x0, 0xdf8475800, 0xa}, {0x3118f00, 0xc0005a2880}) │
│ /__w/cortex/cortex/pkg/ring/kv/dynamodb/dynamodb.go:66 +0x205 │
│ github.com/cortexproject/cortex/pkg/ring/kv/dynamodb.NewClient({{0xc000914f00, 0x9}, {0xc00091c858, 0x11}, 0x0, 0xdf8475800, 0xa}, {0x313d980?, 0xc0005d4738}, {0x3118f00, ...}, ...) │
│ /__w/cortex/cortex/pkg/ring/kv/dynamodb/client.go:59 +0xb2 │
│ github.com/cortexproject/cortex/pkg/ring/kv.createClient({, }, {, _}, {{{0xc000914f00, 0x9}, {0xc00091c858, 0x11}, 0x0, 0xdf8475800, ...}, ...}, ...) │
│ /__w/cortex/cortex/pkg/ring/kv/client.go:139 +0x4c5 │
│ github.com/cortexproject/cortex/pkg/ring/kv.NewClient({{0xc000914f38, 0x8}, {0x2a07581, 0xb}, {{{0xc000914f00, 0x9}, {0xc00091c858, 0x11}, 0x0, 0xdf8475800, ...}, ...}, ...}, ...) │
│ /__w/cortex/cortex/pkg/ring/kv/client.go:130 +0xfa │
│ github.com/cortexproject/cortex/pkg/ring.NewLifecycler({{{{0xc000914f38, 0x8}, {0x2a07581, 0xb}, {{...}, {...}, {...}, {...}, }, {, ...}}, ...}, ...}, ...) │
│ /__w/cortex/cortex/pkg/ring/lifecycler.go:151 +0x2bd │
│ github.com/cortexproject/cortex/pkg/ingester.New({{{{{...}, {...}, {...}, {...}}, _, _, , {, _, _}, ...}, ...}, ...}, ...) │
│ /__w/cortex/cortex/pkg/ingester/ingester.go:647 +0x67a │
│ github.com/cortexproject/cortex/pkg/cortex.(*Cortex).initIngesterService(0xc00065c000) │
│ /__w/cortex/cortex/pkg/cortex/modules.go:413 +0x27d │
│ github.com/cortexproject/cortex/pkg/util/modules.(*Manager).initModule(0xc000748120, {0x7ffe4819df9b, 0x8}, 0x1e?, 0x0?) │
│ /__w/cortex/cortex/pkg/util/modules/modules.go:106 +0x22b │
│ github.com/cortexproject/cortex/pkg/util/modules.(*Manager).InitModuleServices(0x2528260?, {0xc0002e1f90, 0x1, 0x8?}) │
│ /__w/cortex/cortex/pkg/util/modules/modules.go:78 +0xd8 │
│ github.com/cortexproject/cortex/pkg/cortex.(*Cortex).Run(0xc00065c000) │
│ /__w/cortex/cortex/pkg/cortex/cortex.go:408 +0x3bc │
│ main.main() │
│ /__w/cortex/cortex/cmd/cortex/main.go:203 +0xe90

Platform : kubernetes
Installation: Through Helm chart
Image: quay.io/cortexproject/cortex:v1.16.0

Configuration for the Ingestor
ingester:
lifecycler:
# -- We don't want to join immediately, but wait a bit to see other ingesters and their tokens first.
# It can take a while to have the full picture when using gossip
join_after: 10s

  # -- To avoid generating same tokens by multiple ingesters, they can "observe" the ring for a while,
  # after putting their own tokens into it. This is only useful when using gossip, since multiple
  # ingesters joining at the same time can have conflicting tokens if they don't see each other yet.
  observe_period: 10s
  # -- Duration to sleep for before exiting, to ensure metrics are scraped.
  final_sleep: 30s
  num_tokens: 512
  ring:
    # -- Ingester replication factor per default is 3
    replication_factor: 3
    kvstore:
      store: "dynamodb"
      dynamodb:
        region: "us-west-2"
        table_name: "Table_Name"

Also tried by setting the AWS region via the environment. The aws credentials are upload via the serviceaccount, I have used the same serviceaccount with another golang-pod which also talks to the dynamodb and that seem to be working.

Looking for some suggestions

[yeya24]

@danielblando Can you help take a look?

[danielblando]

@kushal-otter

Found the issue. We are setting the config in the wrong configuration. They request us to set in awsConfig and not sessionConfig. I am just not sure why it causes a null pointer in aws-sdk.

I will create a fix for it.

Also tried by setting the AWS region via the environment.

Does this works? This is how we run it and should work. Are you put env var and removing region from config?

[kushal-otter]

@danielblando Thanks for that suggestion. So I was putting the AWS_REGION env variable as well as
dynamodb:
region: "us-west-2"
This was causing the problem naturally, Once I removed the above line and reinstalled, It seems to be working, I will be testing more and keep you updated if I find any issues or problems. Thanks for unblocking me.

[danielblando]

Fix merged
#5842