Peeking from compressed files throws an Exception

[gixxi]

Hi there,

we implement a WAL (write ahead log) using your library. we are facing the issue, that peeking from a queue fails with an exception when the head is within a compressed queue file.

[hewrin]

Below is the error that we receive

2. Unhandled clojure.lang.Compiler$CompilerException
   Error compiling test/lambdaroyal/memory/eviction/test_wal_log_basics.clj at (54:1)

             Compiler.java: 7526  clojure.lang.Compiler/load
                      REPL:    1  user/eval19660
                      REPL:    1  user/eval19660
             Compiler.java: 7062  clojure.lang.Compiler/eval
             Compiler.java: 7025  clojure.lang.Compiler/eval
                  core.clj: 3206  clojure.core/eval
                  core.clj: 3202  clojure.core/eval
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  657  clojure.core/apply
                  core.clj: 1965  clojure.core/with-bindings*
                  core.clj: 1965  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  243  clojure.main/repl/read-eval-print/fn
                  main.clj:  243  clojure.main/repl/read-eval-print
                  main.clj:  261  clojure.main/repl/fn
                  main.clj:  261  clojure.main/repl
                  main.clj:  177  clojure.main/repl
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  202  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  201  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  748  java.lang.Thread/run

1. Caused by java.lang.IndexOutOfBoundsException
   Reader index error. index(8) < 0 || index(8)+length(682) >
   writerIndex(512)

        BytesAbstract.java:  494  io.appulse.utils.BytesAbstract/checkReaderBounds
      BytesFixedArray.java:  181  io.appulse.utils.BytesFixedArray/readBytes
            BytesPool.java:  326  io.appulse.utils.BytesPool$PooledBytes/readBytes
        BytesAbstract.java:  369  io.appulse.utils.BytesAbstract/readBytes
            BytesPool.java:  195  io.appulse.utils.BytesPool$PooledBytes/readBytes
        BytesAbstract.java:  363  io.appulse.utils.BytesAbstract/readBytes
            BytesPool.java:  195  io.appulse.utils.BytesPool$PooledBytes/readBytes
         Deserializer.java:  156  org.infobip.lib.popout.Deserializer$StringDeserializer/deserialize
         Deserializer.java:  151  org.infobip.lib.popout.Deserializer$StringDeserializer/deserialize
    ItemSerialization.java:   51  org.infobip.lib.popout.synced.ItemSerialization/deserialize
      SyncedFileQueue.java:  111  org.infobip.lib.popout.synced.SyncedFileQueue/lambda$peek$2
   ReadWriteBytesPool.java:   66  org.infobip.lib.popout.ReadWriteBytesPool/borrow
      SyncedFileQueue.java:  108  org.infobip.lib.popout.synced.SyncedFileQueue/peek
NativeMethodAccessorImpl.java:   -2  sun.reflect.NativeMethodAccessorImpl/invoke0
NativeMethodAccessorImpl.java:   62  sun.reflect.NativeMethodAccessorImpl/invoke
DelegatingMethodAccessorImpl.java:   43  sun.reflect.DelegatingMethodAccessorImpl/invoke
               Method.java:  498  java.lang.reflect.Method/invoke
            Reflector.java:   93  clojure.lang.Reflector/invokeMatchingMethod
            Reflector.java:  313  clojure.lang.Reflector/invokeNoArgInstanceMember
   test_wal_log_basics.clj:   54  lambdaroyal.memory.eviction.test-wal-log-basics/eval19676
   test_wal_log_basics.clj:   54  lambdaroyal.memory.eviction.test-wal-log-basics/eval19676
             Compiler.java: 7062  clojure.lang.Compiler/eval
             Compiler.java: 7514  clojure.lang.Compiler/load
                      REPL:    1  user/eval19660
                      REPL:    1  user/eval19660
             Compiler.java: 7062  clojure.lang.Compiler/eval
             Compiler.java: 7025  clojure.lang.Compiler/eval
                  core.clj: 3206  clojure.core/eval
                  core.clj: 3202  clojure.core/eval
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn/fn
                  AFn.java:  152  clojure.lang.AFn/applyToHelper
                  AFn.java:  144  clojure.lang.AFn/applyTo
                  core.clj:  657  clojure.core/apply
                  core.clj: 1965  clojure.core/with-bindings*
                  core.clj: 1965  clojure.core/with-bindings*
               RestFn.java:  425  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   87  nrepl.middleware.interruptible-eval/evaluate/fn
                  main.clj:  243  clojure.main/repl/read-eval-print/fn
                  main.clj:  243  clojure.main/repl/read-eval-print
                  main.clj:  261  clojure.main/repl/fn
                  main.clj:  261  clojure.main/repl
                  main.clj:  177  clojure.main/repl
               RestFn.java: 1523  clojure.lang.RestFn/invoke
    interruptible_eval.clj:   84  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:   56  nrepl.middleware.interruptible-eval/evaluate
    interruptible_eval.clj:  152  nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
                  AFn.java:   22  clojure.lang.AFn/run
               session.clj:  202  nrepl.middleware.session/session-exec/main-loop/fn
               session.clj:  201  nrepl.middleware.session/session-exec/main-loop
                  AFn.java:   22  clojure.lang.AFn/run
               Thread.java:  748  java.lang.Thread/run

[xxlabaza]

hi! can you please provide a code example? I'm going to reproduce it in the tests and fix the issue

[hewrin]

Hi, here is our code example

https://gist.github.com/hewrin/cdc4ab885f0e7bc09716f4e5291a115f

Attached also are the wal files that cause the issue.

wal2.zip

Thanks in advance

[gixxi]

Hi Artem,

are you able to reproduce the issue, can we somehow support you providing a bugfix.

thanks in advance & kind regards

[aerovulpe]

I also had this issue. Flushing the queue before calling peek resolved it for me. I hope this helps anyone coming across this in the future.