calling d[:history] in a for loop

[EthanRLeonard]

Appreciate any time you can give to what's probably a simple issue.

Jib.disconnect(ib)
symbols = ["EUR","CHF","NZD"]
shelves = Any[]
ib = Jib.connect(4002,22);
d,w2 = Jib.simple_wrap();
Jib.start_reader(ib,w2);
for (idx, s) in enumerate(symbols)
local contract2 = Jib.Contract(symbol=s, secType="CASH", exchange="IDEALPRO", currency="USD")
Jib.reqHistoricalData(ib,100*idx,contract2, "00:00:00 US/Eastern", "1 D", "10 mins", "MIDPOINT", true, 1, false)
d[:history]
hd = d[:history]
push!(shelves,(hd[:,"close"] .- hd[:,"open"]))
end

This gives

Error: -1Load Error:
Key Error: key 2104
Market data farm connection is OK:usfarm.nj
error: -1 2104 Market data farm connection is OK:usfuture
error: -1 2104 Market data farm connection is OK:cashfarm
error: -1 2104 Market data farm connection is OK:usfarm
error: -1 2106 HMDS data farm connection is OK:cashhmds
error: -1 2106 HMDS data farm connection is OK:ushmds
error: -1 2158 Sec-def data farm connection is OK:secdefil
:history not found

The REPL is kept open however, and if I type d, it gives the nextID, accounts, and history outputs, for the first symbol.
The symbols work individually if I type in reqHistoricalData() and then d[:history] sequentially. How can I make it run in a for loop?

Benefited a lot from this answer of yours:
#4 (comment)

This library is pretty awesome, and I'm really appreciating how thorough everything is and how you have responded to new users. Thanks a lot.

[lbilli]

The main issue is that IB API is designed around an asynchronous communication model: i.e. conceptually, sending and receiving data are handled by separate processes which are not in sync.

In particular, functions that send requests return immediately, without waiting for responses to arrive.
Responses, if any, arrive at unpredictable times and are handled in a separate line of execution ("reader process").

Some general references on the topic from Julia docs and the IB API docs.

In your example, the attempt to access d[history] happens right after sending the request, when the requested data has not arrived yet (in fact the request itself may not even have left your computer!).

A quick and dirty, certainly frowned upon and ultimately wrong fix is to insert a sleep(...) just before hd = d[history]. That would halt the execution of the program giving hopefully enough time to the data to arrive.

In general, the coordination between processes requires more planning and a different mindset compared to the usual deterministic sequential execution and involves the use of tools like wait, notify, Condition and Channel.

An quick alternative to your example (which still involves a wait at the end though):

using DataFrames, Jib

symbols = ["EUR", "CHF", "NZD"]
history = Dict{String,DataFrame}()

wrap = Jib.Wrapper(

      # Save history
      historicalData= function(reqId, bar)
              println("history $reqId")
              history[symbols[reqId]] = bar
            end,

      # Boilerplate methods
      error= (id, errorCode, errorString, advancedOrderRejectJson) ->
            println("Error: $(something(id, "NA")) $errorCode $errorString $advancedOrderRejectJson"),

      nextValidId= (orderId) -> println("Next OrderId: $orderId"),

      managedAccounts= (accountsList) -> println("Managed Accounts: $accountsList")
      )

ib = Jib.connect(4002,22)

Jib.start_reader(ib, wrap)

for (idx, s) in enumerate(symbols)
  contract = Jib.Contract(symbol=s, secType="CASH", exchange="IDEALPRO", currency="USD")
  Jib.reqHistoricalData(ib, idx, contract, "", "1 D", "10 mins", "MIDPOINT", true, 1, false)
end

# Wait up to 10s
res = timedwait(() -> length(history) == length(symbols), 10) 

if res == :ok
  # All data has arrived
  history["EUR"]
  history["CHF"]
  # ...
else
  # Data has not arrived, maybe needs more time...
  # Do something else
end

[EthanRLeonard]

Amazing. Thank you. Really appreciate you explaining this carefully and with a working example.