Request Mask Filter for Actions

[gktval]

Great project, thanks for sharing. It would be great to have more flexibility between the agent and the environment. For instance, many environments mask out the actions by setting them to int.MinValue.

Would it be possible to return the array of all actions before the argmax? Then add a method for the iEnvironment, something like MaskActions(actionIds)?

[gktval]

I think I figured out a way to handle this. I create a variable to hold the action masks in ComposableQDiscreteAgent, then expose it in the ppo/dqn agents. Prior to the selection, it multiplies the action probabilities with the masks, then does the selection.

[bnemetchek2]

@gktval , could you explain this a little more or show a short example of how you did this? I am also interested in how to implement action validation masks.

[gktval]

@bnemetchek2 I will try and get this pushed up. Busy week.

[bnemetchek2]

@gktval I also agree with your sentiment that this is a very nice project. I'm surprised at how few downloads it has so far. I watched the youtube on it a few months back it looked like a bit of a learning curve but after recently starting to use it, I'm very impressed. I think more documentation and updated examples would go a long ways. to improving the reception. Also, thanks the help the mask stuff. I shall await patiently...

[asieradzk]

@bnemetchek2 I'm working on documentation recently. Its incredibly time consuming! Did you have a chance to check it out so far at https://www.rlmatrix.com/

[bnemetchek2]

@asieradzk I had not seen the documentation site. I had seen you had a project for it but didn't know where it was. A smarter person than I would have guessed what is was. Maybe throw a reference to it on the main project's github page? Thanks again for this project. I've watched parts of your youtube video several times now and get more out of it each time. They really made you rush over a lot of stuff. The dashboard is also very useful and appreciated.

[gktval]

I just realized I have lots of changes that I don't know if @asieradzk wants. I was using the library for two competing agents.

List of my changes:
A property for keeping track of the agent
Changed the memory replay to a queue in order to fix capacity that is being ignored (ie, memory leak)
Changed Step() to return Task<int> instead of Task so that I could keep track of the selected action
In the example environments, the GetCurrentState() reset the environment. I only wanted to reset the environment at the end of the step. Therefore I updated the Step() so that I could get the pre and post state of the environment without resetting it on the first GetStateAsync()
Finally, the ability to pass in ActionsMask(int[] mask)

I don't feel comfortable pulling to this repository, but feel free to cherry pick whatever you want out of mine:
https://github.com/gktval/RL_Matrix

Take a look at https://github.com/gktval/RL_Matrix/blob/master/src/RLMatrix/Agents/Common/LocalDiscreteRolloutAgent.cs to get a feel for the ActionMasks.