Code

Memmap shuffling #216

Code

Open

Lewington-pitsos wants to merge 24 commits intojbloomAus:mainjbloomAus/SAELens:mainfrom Lewington-pitsos:memmap_shufflingLewington-pitsos/SAELens:memmap_shufflingCopy head branch name to clipboard

+459-410

Commits

Commits on Jul 2, 2024

• call dataloader next consistent with async loading

  jbloomAus

  committed
  ee78d91

  Copy full SHA for ee78d91

• start using np.memmap (but with none of the actual advantages

  jbloomAus

  committed
  eda7db7

  Copy full SHA for eda7db7

Commits on Jul 6, 2024

• various changes, iterating toward np.memmap

  jbloomAus

  committed
  bff803f

  Copy full SHA for bff803f

• deduplicate small mathematical operations

  Lewington-pitsos

  committed
  c486f05

  Copy full SHA for c486f05

• named parameters for shuffle idxs

  Lewington-pitsos

  committed
  112960a

  Copy full SHA for 112960a

• add diagnostic prints, fix typing in activations_store

  Lewington-pitsos

  committed
  e01238d

  Copy full SHA for e01238d

• add diagnostic prints, fix typing in activations_store

  Lewington-pitsos

  committed
  545a33c

  Copy full SHA for 545a33c

• rename shuffling methods

  Lewington-pitsos

  committed
  7eec1c2

  Copy full SHA for 7eec1c2

Commits on Jul 7, 2024

• add dataset override for cache activations runner, update test to use small custom dataset

  Lewington-pitsos

  committed
  a720ad0

  Copy full SHA for a720ad0

• replicate error using activationstore alone

  Lewington-pitsos

  committed
  18f4216

  Copy full SHA for 18f4216

• replicate error using activationstore alone

  Lewington-pitsos

  committed
  c053910

  Copy full SHA for c053910

• fix float32 vs float16 memmap double size issue

  Lewington-pitsos

  committed
  a76e4ed

  Copy full SHA for a76e4ed

• fix typing

  Lewington-pitsos

  committed
  4b7fbdb

  Copy full SHA for 4b7fbdb

• skip test_load_cached_activations

  Lewington-pitsos

  committed
  bf3e28d

  Copy full SHA for bf3e28d

• get all unit test passing with new next_batch functionality

  Lewington-pitsos

  committed
  7a74cf6

  Copy full SHA for 7a74cf6

• Merge branch 'main' into memmap_shuffling

  Lewington-pitsos

  committed
  9ca4aba

  Copy full SHA for 9ca4aba

• merge with main

  Lewington-pitsos

  committed
  43ccbb8

  Copy full SHA for 43ccbb8

• format

  Lewington-pitsos

  committed
  d82b17c

  Copy full SHA for d82b17c

• add small test of next_batch functionality

  Lewington-pitsos

  committed
  a955a7f

  Copy full SHA for a955a7f

• format again

  Lewington-pitsos

  committed
  ac45f4c

  Copy full SHA for ac45f4c

• map bfloat16 to np.float32

  Lewington-pitsos

  committed
  c80de83

  Copy full SHA for c80de83

• reformat

  

  Your Name

  committed
  fc147d5

  Copy full SHA for fc147d5

• add memory pinning

  

  Your Name

  committed
  4e8847b

  Copy full SHA for 4e8847b

Commits on Jul 8, 2024

• reformat

  Lewington-pitsos

  committed
  fd0d499

  Copy full SHA for fd0d499