Skip to content

Commit

Permalink
QFJ-950 reject garbled messages (#206)
Browse files Browse the repository at this point in the history 
QFJ-950: Provide configuration to have garbled messages rejected instead of ignoring them
- added configuration to Session class
- Update documentation, add explanation for settings RejectInvalidMessage and RejectGarbledMessage
- extended InvalidMessage to optionally pass problematic message for later processing
- Introduced method MessageUtils.newInvalidMessageException().
- Change Session and AbstractIoHandler to process garbled messages when configuration is enabled.
  • Loading branch information
@chrjohn
chrjohn authored Aug 2, 2018
1 parent c790dac commit cc7632f
Show file tree
Hide file tree
Showing 12 changed files with 458 additions and 63 deletions.
91 changes: 90 additions & 1 deletion quickfixj-core/src/main/doc/usermanual/usage/configuration.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ <H3>QuickFIX Settings</H3>
<LI><A HREF="#Storage">Storage</A></LI>
<LI><A HREF="#Logging">Logging</A></LI>
<LI><A HREF="#Miscellaneous">Miscellaneous</A></LI>
<LI><A HREF="#Invalid vs Garbled Messages">Invalid vs Garbled Messages</A></LI>
<LI><A HREF="#Sample Settings File">Sample Settings File</A></LI>
</UL>
<TABLE class="settings" cellspacing="0">
Expand Down Expand Up @@ -420,9 +421,21 @@ <H3>QuickFIX Settings</H3>
<TD> 120 </TD>

</TR>
<TR ALIGN="left" VALIGN="middle">
<TD> <I>RejectGarbledMessage</I> </TD>
<TD> If RejectGarbledMessage is set to Y, garbled messages will be rejected (with a generic error message in 58/Text field) instead of ignored.<br>
This is only working for messages that pass the FIX decoder and reach the engine.<br>
Messages that cannot be considered a real FIX message (i.e. not starting with 8=FIX or not ending with 10=xxx) will be ignored in any case.<br>
See <A HREF="#Invalid vs Garbled Messages">Invalid vs Garbled Messages</A> for further explanation.
</TD>
<TD> Y<br>N</TD>
<TD> N </TD>
</TR>
<TR ALIGN="left" VALIGN="middle">
<TD> <I>RejectInvalidMessage</I> </TD>
<TD> If RejectInvalidMessage is set to N, only a warning will be logged on reception of message that fails data dictionary validation.</TD>
<TD> If RejectInvalidMessage is set to N, only a warning will be logged on reception of message that fails data dictionary validation.<br>
See <A HREF="#Invalid vs Garbled Messages">Invalid vs Garbled Messages</A> for further explanation.
</TD>
<TD> Y<br>N</TD>
<TD> Y </TD>
</TR>
Expand Down Expand Up @@ -1251,6 +1264,82 @@ <H3>QuickFIX Settings</H3>
</tbody>
</TABLE>

<H3><A NAME="Invalid vs Garbled Messages">Rejecting Invalid vs Garbled Messages</A></H3>

<p>
There are mainly two settings that influence QFJ's rejection behaviour:
</p>
<ul>
<li>RejectInvalidMessage</li>
<li>RejectGarbledMessage</li>
</ul>

<p>
While the first applies to messages that fail data dictionary validation,
the latter applies to messages that fail basic validity checks on the FIX protocol level.
</p>

<H4>Setting RejectInvalidMessage</H4>

<p>
If RejectInvalidMessage is set to
</p>
<ul>
<li>Y, the problematic message will be rejected (this is the default setting).</li>
<li>N, only a warning will be logged on reception of a message that fails data dictionary validation.
The message will then be handed over to the application level code.</li>
</ul>

<H4>Setting RejectGarbledMessage</H4>

<p>
If RejectGarbledMessage is set to
</p>
<ul>
<li>Y, garbled messages will be rejected (with a generic error message in 58/Text field) instead of ignored.</li>
<li>N, garbled messages will be ignored and the sequence number not be incremented (this is the default setting).</li>
</ul>

<H5>Information on garbled messages</H5>
<p>
In FIX it is legal to ignore a message under certain circumstances. Since FIX is an optimistic protocol
it expects that some errors are transient and will correct themselves with the next message transmission.
Therefore the sequence number is not incremented and a resend request is issued on the next received
message that has a higher sequence number than expected.
</p>

<p>
In the case that the error is not transient, the default behaviour is not optimal because not consuming a
message sequence number can lead to follow-up problems since QFJ will wait for the message to be resent
and queue all subsequent messages until the resend request has been satisfied (i.e. infinite resend loop).
</p>

What constitutes a garbled message (taken from the FIX protocol specification):
<blockquote>
<li>BeginString (tag #8) is not the first tag in a message or is not of the format 8=FIXT.n.m.</li>
<li>BodyLength (tag #9) is not the second tag in a message or does not contain the correct byte count.</li>
<li>MsgType (tag #35) is not the third tag in a message.</li>
<li>Checksum (tag #10) is not the last tag or contains an incorrect value.</li>

If the MsgSeqNum(tag #34) is missing a logout message should be sent terminating the FIX Connection, as this
indicates a serious application error that is likely only circumvented by software modification.
</blockquote>

<p>
You have the possibility to adapt QFJ's behaviour for some of the cases mentioned above.<br>
</p>
<li>If an incoming message does neither start with the BeginString tag nor does it end with the Checksum tag, the message
cannot be passed to the session and will be discarded by the FIX decoder right away.</li>
<li>Examples where the message will be rejected instead of ignored when RejectGarbledMessage=Y:
<ul>
<li>incorrect checksum</li>
<li>repeating group count field contains no valid integer</li>
<li>no SOH delimiter found in field</li>
<li>missing MsgType</li>
<li>invalid tags, e.g. 49foo=bar</li>
</ul>
</li>

<H3><A NAME="Sample Settings File">Sample Settings File</A></H3>

<p>
Expand Down
5 changes: 4 additions & 1 deletion quickfixj-core/src/main/java/quickfix/DefaultSessionFactory.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,9 @@ public Session create(SessionID sessionID, SessionSettings settings) throws Conf
try {
String connectionType = null;

final boolean rejectGarbledMessage = getSetting(settings, sessionID,
Session.SETTING_REJECT_GARBLED_MESSAGE, false);

final boolean rejectInvalidMessage = getSetting(settings, sessionID,
Session.SETTING_REJECT_INVALID_MESSAGE, true);

Expand Down Expand Up @@ -209,7 +212,7 @@ public Session create(SessionID sessionID, SessionSettings settings) throws Conf
resetOnLogon, resetOnLogout, resetOnDisconnect, refreshAtLogon, checkCompID,
redundantResentRequestAllowed, persistMessages, useClosedIntervalForResend,
testRequestDelayMultiplier, senderDefaultApplVerID, validateSequenceNumbers,
logonIntervals, resetOnError, disconnectOnError, disableHeartBeatCheck,
logonIntervals, resetOnError, disconnectOnError, disableHeartBeatCheck, rejectGarbledMessage,
rejectInvalidMessage, rejectMessageOnUnhandledException, requiresOrigSendingTime,
forceResendWhenCorruptedStore, allowedRemoteAddresses, validateIncomingMessage,
resendRequestChunkSize, enableNextExpectedMsgSeqNum, enableLastMsgSeqNumProcessed);
Expand Down
26 changes: 26 additions & 0 deletions quickfixj-core/src/main/java/quickfix/InvalidMessage.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,16 +25,42 @@
*/
public class InvalidMessage extends Exception {

Message fixMessage;

public InvalidMessage() {
super();
}

public InvalidMessage(Message fixMessage) {
super();
setGarbledFixMessage(fixMessage);
}

public InvalidMessage(String message) {
super(message);
}

public InvalidMessage(String message, Message fixMessage) {
super(message);
setGarbledFixMessage(fixMessage);
}

public InvalidMessage(String message, Throwable cause) {
super(message, cause);
}

public InvalidMessage(String message, Throwable cause, Message fixMessage) {
super(message, cause);
setGarbledFixMessage(fixMessage);
}

public Message getFixMessage() {
return fixMessage;
}

private void setGarbledFixMessage(Message fixMessage) {
this.fixMessage = fixMessage;
this.fixMessage.setGarbled(true);
}

}
32 changes: 19 additions & 13 deletions quickfixj-core/src/main/java/quickfix/Message.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -563,11 +563,11 @@ private void validateCheckSum(String messageData) throws InvalidMessage {
final int checksum = trailer.getInt(CheckSum.FIELD);
if (checksum != MessageUtils.checksum(messageData)) {
// message will be ignored if checksum is wrong or missing
throw new InvalidMessage("Expected CheckSum=" + MessageUtils.checksum(messageData)
+ ", Received CheckSum=" + checksum + " in " + messageData);
throw MessageUtils.newInvalidMessageException("Expected CheckSum=" + MessageUtils.checksum(messageData)
+ ", Received CheckSum=" + checksum + " in " + messageData, this);
}
} catch (final FieldNotFound e) {
throw new InvalidMessage("Field not found: " + e.field + " in " + messageData);
throw MessageUtils.newInvalidMessageException("Field not found: " + e.field + " in " + messageData, this);
}
}

Expand All @@ -579,7 +579,7 @@ && isNextField(dd, header, BodyLength.FIELD)
if (!validHeaderFieldOrder) {
// Invalid message preamble (first three fields) is a serious
// condition and is handled differently from other message parsing errors.
throw new InvalidMessage("Header fields out of order in " + messageData);
throw MessageUtils.newInvalidMessageException("Header fields out of order in " + messageData, MessageUtils.getMinimalMessage(messageData));
}
}

Expand Down Expand Up @@ -609,7 +609,7 @@ private String getMsgType() throws InvalidMessage {
try {
return header.getString(MsgType.FIELD);
} catch (final FieldNotFound e) {
throw new InvalidMessage(e.getMessage() + " in " + messageData);
throw MessageUtils.newInvalidMessageException(e.getMessage() + " in " + messageData, this);
}
}

Expand Down Expand Up @@ -663,7 +663,7 @@ private void parseGroup(String msgType, StringField field, DataDictionary dd, Da
try {
declaredGroupCount = Integer.parseInt(field.getValue());
} catch (final NumberFormatException e) {
throw new InvalidMessage("Repeating group count requires an Integer but found: " + field.getValue(), e);
throw MessageUtils.newInvalidMessageException("Repeating group count requires an Integer but found '" + field.getValue() + "' in " + messageData, this);
}
parent.setField(groupCountTag, field);
final int firstField = rg.getDelimiterField();
Expand Down Expand Up @@ -828,10 +828,9 @@ static boolean isTrailerField(int field) {
// Extract field
//
private String messageData;

private int position;

private StringField pushedBackField;
private boolean isGarbled = false;

public void pushBack(StringField field) {
pushedBackField = field;
Expand All @@ -851,20 +850,20 @@ private StringField extractField(DataDictionary dataDictionary, FieldMap fields)

final int equalsOffset = messageData.indexOf('=', position);
if (equalsOffset == -1) {
throw new InvalidMessage("Equal sign not found in field" + " in " + messageData);
throw MessageUtils.newInvalidMessageException("Equal sign not found in field in " + messageData, this);
}

int tag;
try {
tag = Integer.parseInt(messageData.substring(position, equalsOffset));
} catch (final NumberFormatException e) {
position = messageData.indexOf('\001', position + 1) + 1;
throw new InvalidMessage("Bad tag format: " + e.getMessage() + " in " + messageData);
throw MessageUtils.newInvalidMessageException("Bad tag format: " + e.getMessage() + " in " + messageData, this);
}

int sohOffset = messageData.indexOf('\001', equalsOffset + 1);
if (sohOffset == -1) {
throw new InvalidMessage("SOH not found at end of field: " + tag + " in " + messageData);
throw MessageUtils.newInvalidMessageException("SOH not found at end of field: " + tag + " in " + messageData, this);
}

if (dataDictionary != null && dataDictionary.isDataField(tag)) {
Expand All @@ -878,7 +877,7 @@ private StringField extractField(DataDictionary dataDictionary, FieldMap fields)
try {
fieldLength = fields.getInt(lengthField);
} catch (final FieldNotFound e) {
throw new InvalidMessage("Did not find length field " + e.field + " required to parse data field " + tag + " in " + messageData);
throw MessageUtils.newInvalidMessageException("Did not find length field " + e.field + " required to parse data field " + tag + " in " + messageData, this);
}

// since length is in bytes but data is a string, and it may also contain an SOH,
Expand All @@ -889,7 +888,7 @@ private StringField extractField(DataDictionary dataDictionary, FieldMap fields)
&& messageData.substring(equalsOffset + 1, sohOffset).getBytes(CharsetSupport.getCharsetInstance()).length < fieldLength) {
sohOffset = messageData.indexOf('\001', sohOffset + 1);
if (sohOffset == -1) {
throw new InvalidMessage("SOH not found at end of field: " + tag + " in " + messageData);
throw MessageUtils.newInvalidMessageException("SOH not found at end of field: " + tag + " in " + messageData, this);
}
}
}
Expand Down Expand Up @@ -936,5 +935,12 @@ public static MsgType identifyType(String message) throws MessageParseError {
}
}

boolean isGarbled() {
return isGarbled;
}

void setGarbled(boolean isGarbled) {
this.isGarbled = isGarbled;
}

}
41 changes: 38 additions & 3 deletions quickfixj-core/src/main/java/quickfix/MessageUtils.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ private static String getFieldOrDefault(FieldMap fields, int tag, String default
}

/**
* Utility method for parsing a mesasge. This should only be used for parsing messages from
* Utility method for parsing a message. This should only be used for parsing messages from
* FIX versions 4.4 or earlier.
*
* @param messageFactory
Expand Down Expand Up @@ -175,7 +175,7 @@ private static ApplVerID getApplVerID(Session session, String messageString)
}

if (applVerID == null) {
throw new InvalidMessage("Can't determine ApplVerID for message");
throw newInvalidMessageException("Can't determine ApplVerID from message " + messageString, getMinimalMessage(messageString));
}

return applVerID;
Expand Down Expand Up @@ -204,11 +204,32 @@ private static boolean isMessageType(String message, String msgType) {
public static String getMessageType(String messageString) throws InvalidMessage {
final String value = getStringField(messageString, 35);
if (value == null) {
throw new InvalidMessage("Missing or garbled message type in " + messageString);
throw newInvalidMessageException("Missing or garbled message type in " + messageString, getMinimalMessage(messageString));
}
return value;
}

/**
* Tries to set MsgSeqNum and MsgType from a FIX string to a new Message.
* These fields are referenced on the outgoing Reject message.
*
* @param messageString FIX message as String
* @return New quickfix.Message with optionally set header fields MsgSeqNum
* and MsgType.
*/
static Message getMinimalMessage(String messageString) {
final Message tempMessage = new Message();
final String seqNum = getStringField(messageString, 34);
if (seqNum != null) {
tempMessage.getHeader().setString(34, seqNum);
}
final String msgType = getStringField(messageString, 35);
if (msgType != null) {
tempMessage.getHeader().setString(35, msgType);
}
return tempMessage;
}

public static String getStringField(String messageString, int tag) {
String value = null;
final String tagString = Integer.toString(tag);
Expand Down Expand Up @@ -362,4 +383,18 @@ public static int checksum(String message) {
public static int length(Charset charset, String data) {
return CharsetSupport.isStringEquivalent(charset) ? data.length() : data.getBytes(charset).length;
}

/**
* Returns an InvalidMessage Exception with optionally attached FIX message.
*
* @param errorMessage error description
* @param fixMessage problematic FIX message
* @return InvalidMessage Exception
*/
static InvalidMessage newInvalidMessageException(String errorMessage, Message fixMessage) {
if (fixMessage != null) {
return new InvalidMessage(errorMessage, fixMessage);
}
return new InvalidMessage(errorMessage);
}
}
Loading

0 comments on commit cc7632f

Please sign in to comment.