DaysOnMarket Client Auto-increment Proposal

[bryanburgers]

The Days on Market auto-increment proposal is a proposal that adds three new fields to the Property resource of the Data Dictionary, and defines how those three fields work together to help MLSes, data providers, and consumers stay in sync with respect to Days on Market.

Key points of emphasis

There are two key points of emphasis regarding this proposal.

1. The MLS is still in control of Days on Market.
2. The proposal solves long-standing Days on Market replication issues.

Point 1. This proposal does not try to standardize how the MLS calculates Days on Market nor does it require the MLS to share their method of calculating Days on Market or require the MLS to share all of the data points that go into calculating the Days on Market. The MLS remains the source of truth for Days on Market.

Point 2. This proposal does require that when an MLS provides a Days on Market value, it also provides an associated reference timestamp. By using these two values, consumers are able to interpolate the Days on Market value in the time between updates to a listing.

Age and record keeping

How old are you? It's a question you probably know the answer to. In a way, your age is one of data points that make up who you are.

And when communicating with humans, telling someone your age (although sometimes taboo) is far more convenient than telling them what year you were born.

That said, you likely have a driver's license or an identity card of some sort. If you look at it, you won't find your age written anywhere on it. Why not? Because if it was, you'd have to get a new one every year.

Instead, what you will see is your birthdate, a reference date that can be used to calculate your age no matter what day it is.

When we store and retrieve records, it's typically more accurate to store data points independent of time.

Days on Market is more complicated than your age, but the idea for this proposal is the same: instead of giving out an age (the exact number of days a listing has been on market) at a particular point in time, this proposal provides a way to give out three pieces of reference data that make it easy to increment the number Days on Market since the last data fetch no matter how long it has been since that data fetch.

The new fields

Field	Type	Description
DomIncrementBase	Number	Used in the calculation of the number of days the listing is on market as defined by the MLS business rules, the number to use as the reference point in the increment calculation.
DomIncrementTimestamp	Timestamp	Used in the calculation of the number of days the listing is on market as defined by the MLS business rules, the timestamp associated with the base reference point.
DomIncrementYN	Boolean	Used in the calculation of the number of days the listing is on market as defined by MLS business rules, whether the increment calculation should be run for this listing.

How it works

Whenever a listing gets updated via an API call (whenever the ModificationTimestamp changes), the MLS provides three values that represent the instantaneous Days on Market at that point:

• DomIncrementBase – a number of days that form the base of the increment calculation; typically the number of days on market at the point the listing was updated.
• DomIncrementTimestamp – a timestamp to use when calculating how many days to add to the base number.
• DomIncrementYN – whether the listing should have an incrementing Days on Market.

Every time a listing is updated in the source system, these three values also get updated, giving the consumer an up-to-date view of Days on Market.

Between listing updates – especially if several days elapse between listing updates – the consumer is able to increment the number of Days on Market using the three values to stay in sync with the MLS.

The constant case is when DomIncrementYN is false (this is common for Closed listings, but may also be false for Pending listings depending on MLS Rules). In this case, Days on Market is DomIncrementBase. (It's constant with respect time time.)

The incrementing case is when DomIncrementYN is true (this is common for Active listings). In this case, for a consumer to display the same value as what the MLS displays, the consumer must calculate the number of full days since DomIncrementTimestamp (e.g. if DomIncrementTimestamp is more than 3 days ago but less than 4 days ago, this difference is 3), and add value to the DomIncrementBase.

Further reading

• The original proposal: https://gist.github.com/bryanburgers/c74870723e81f2d4d9e0404123f469b9
• Post in DD General Discussions Confluence: https://reso.atlassian.net/wiki/spaces/DDGDF/pages/9526968321/DaysOnMarket+calculation+proposal
• The problem with Days on Market, a narrative: DaysOnMarket Client Auto-increment Proposal #140 (comment)
• Zenlist DaysOnMarket case studies: https://gist.github.com/bryanburgers/8a450ac2ec584e80c1226163ae234ee8

Example

We'll start with a new listing, listed by an agent on April 21. According to the local rules of the MLS in this example, the listing starts with a Days on Market value of 1.

Field	Value	Comments
DomIncrementYN	true	This is an active listing, so the Days on Market value will increase every day.
DomIncrementBase	1	The base value for performing the calculation. On the day the listing was lsited, Days on Market is 1.
DomIncrementTimestamp	2024-04-21T00:00-07:00	The earliest time at which Days on Market is 1. Consumers wil use this timestamp to calculate the actual Days on Market as we move forward in time. Note: this timestamp does not need to be the same as ModificationTimestamp, and in this case it is midnight local time of the day represented by DomIncrementBase.

On the day that the listing is posted, we can use the calculation to determine the current Days on Market. We start our calculation with the base, which is 1 according to DomIncrementBase. And then we add the total number of full days between DomIncrementTimestamp and now. The day the listing was posted, this value is 0. So 1 + 0 is 1.

The next day, April 22, the consumer hasn't received another listing update yet. So the base is still 1 according to DomIncrementBase. However, on April 22, the number of full days since DomIncrementTimestamp is now 1, so the calculation yields 1 + 1 = 2.

Until the consumer recives another listing update, the calculation continues on. On April 23, the calculation yields 1 + 2 = 3. Note that we're essentially incrementing Days on Market by 1 each day until we receive more information from the MLS.

At the start of April 24, we still haven't received a listing update, so our calculation continues to yield DomIncrementBase (1) + days since DomIncrementTimestamp (3) = 4.

Later on April 24, an agent makes an update to the listing's price. This causes the listing to be updated, which causes the consumer to refetch the listing and get an updated listing payload.
At this point, DomIncrementBase, DomIncrementTimestamp, and DomIncrementYN get new values.

Field	Value	Comments
DomIncrementYN	true	The price change didn't change the fact the listing is active and Days on Market will increase every day.
DomIncrementBase	4	According to the MLS, this listing has been on market for 4 days.
DomIncrmentTimestamp	2024-04-24T00:00-07:00	This is the earliest conceptual time at which the Days on Market was 4. Once again, note that because this MLS increments Days on Market at midnight local time, this is set to midnight local time.

With the new values, on April 24, there are 0 full days between now and DomIncrementTimestamp so days on Market is 4 + 0 = 4. Note that although the values for these fields changed, the resulting calculation did not. Both before and after the update, the value the consumer calculates is 4.

At this point, April 25 shouldn't be a surprise.

On April 26, the listing changes status to pending. The local rules for the MLS in this example dicate that Days on Market does not increase while a listing is in the pending status. Because of that, the values returned are

Field	Value	Comments
DomIncrementYN	false	This value is now false which indicates that Days on Market will not increase every day.
DomIncrementBase	6	This is the number of Days on Market that the listing will continue to have as long as DomIncrementYN is false.
DomIncrementTimestamp	2024-04-26T00:00-07:00	This field is now essentially unused. The MLS in this example chooses to set it for consistency.

For the entire rest of April 26, Days on Market is equal to DomIncrementBase.

On April 27, Days on Market is also equal to DomIncrementBase. No extra days are added to the value, because DomIncrementYN is false.

Days on Market will continue to be 6 until we receive a new listing update that indicates otherwise.

On April 28, the listing is still in a pending state and DomIncrementYN is still false, so the Days on Market is still 6.

Later on April 28, the listing comes back on market. The Days on Market values in the listing update include:

Field	Value	Comments
DomIncrementYN	true	The listing will now increment its Days on Market every day again.
DomIncrementBase	6	The number of Days on Market when the listing was updated is 6.
DomIncrementTimestamp	2024-04-28T00:00-07:00	Consumers can use this updated timestamp to calculate Dasy on Market.

On April 28 after the update, there are 0 full days between DomIncrementTimestamp and now, so Days on Market is 6 + 0 = 6.

On April 29 at Midnight local time, there is 1 full day between DomIncrementTimestamp and now, so Days on Market is 6 + 1 = 7.

On April 30 at Midnight local time, there are 2 full days between DomIncrementTimestamp and now, so days on Market is 6 + 2 = 8.

[bryanburgers]

It sounds like discussion on this may be moved to transport, so here's the original write-up I added to the Data Dictionary General Discussion Forum, so it's in one place.

---

I’ve read through the various DOM discussions in this space and would like to add to the discussion.

I have a proposal that

• Efficiently keeps MLSes and API users in sync regarding DOM (no extra/daily API requests)
• Does not require RESO to standardize the rules for calculating DOM
• Does not require MLSes to communicate via the API their individual rules for calculating DOM

The proposal adds two fields to DD

• DaysOnMarket – existing field
• DaysOnMarketAccumulatingYN – new boolean field that indicates whether the listing, in its current state, is incrementing
• DaysOnMarketRelativeTimestamp – new timestamp field that provides a base from which to calculate DOM over time.

And defines the calculation for “actual/current DOM” as

if DaysOnMarketAccumulatingYN is false:
  current = DaysOnMarket
if DaysOnMarketAccumulatingYN is true:
  current = DaysOnMarket + number of full days between DaysOnMarketRelativeTimestamp and now

The full proposal is currently here: https://gist.github.com/bryanburgers/c74870723e81f2d4d9e0404123f469b9

We’ve been using this internally for a while and we have found it to be effective and general across the MLSes we work with.

I’d love to hear your thoughts and questions on it!

[bryanburgers]

A summary of discussion on the Data Dictionary General Dicussion Forum

• Trestle already does this, using these fields
  • DaysOnMarketReplication
  • DaysOnMarketReplicationDate
  • DaysOnMarketReplicationIncreasingYN
• Ivaan suggested adding three new fields and leaving DaysOnMarket untouched, similiar to how Trestle has DaysOnMarket and DaysOnMarketReplication
• Some discussion on whether the baseline field (DaysOnMarketRelativeTimestamp, DaysOnMarketReplicationDate) should be a Timestamp or a Date

[cobogeri]

Elegant work - and the only comment I have is centered around the "Must" include on DaysOnMarketAccumulatingYN field. What happens when the client doesn't get DaysOnMarket? We have a couple custom clients that don't get the DaysOnMarket field.

I would suggest the "Must" include for DaysOnMarketAccumulatingYN be dependent on the existence DaysOnMarket in the client's feed.

[bryanburgers]

Sounds good. Yes, this proposal is only relevant when an MLS is intentionally sharing DaysOnMarket with a consumer. If that's not the case, the proposal is completely irrelevant.

I'm not clear on the next steps; if it is to write out a Transport spec, I think that spec would need to follow a slightly different format. Specifying in that spec when this scheme does and doesn't apply makes sense. Thanks!

[darnjo]

After the discussion has been reviewed and approved by Transport, the next step is to create a ticket and proposal.

[bryanburgers]

The primary takeaway from the September 2024 Transport meeting is that this proposal will change from using the existing DaysOnMarket field and adding two new fields. Instead, the proposal will add three new fields, and leave the existing DaysOnMarket unchanged. This will allow MLSes and providers that calculate DaysOnMarket at query time and want to continue to do that for backward-compatibility reasons to use the three new fields in the way they're intended.

(Hopefully that captures the discussion accurately.)

I will update the proposal in the next couple of days.

Update: I was asked after the meeting not to follow up on the action item I was given to update the proposal.

[darnjo]

Just to clarify, no decisions or conclusions have been made in the workgroups yet.

RESO will update this issue with the outcome once a recommendation has been reached.

[StevenMayhue]

This seems to me an insidious change in the RESO community.
Creating artificial values that could be used in place of factual data to lessen the workload of the consumer may only be the beginning of such data masking requests. The reason we are all working together is to enable sharing of curated data at scale with assurance of validity.
This introduces the possibility of incorrect data into systems that live and die by the validity of the data in their custodial care. What prevents a consumer from simply using last data snapshot with these artificial values? If a particular API they pull from is problematic then a correction of that endpoint is deserved, but the answer is not to introduce calculated values at the expense of our reputations as good and faithful custodians of the factual data we share.
I fear that even moving this to separate fields may not prevent cross contamination across platforms.

[bryanburgers]

Thank you for your candid feedback!

First of all, I want to point out that we’re on the same page here. I also absolutely believe that the MLS is responsible for accurate data, and that it’s in both the MLS’s and the consumer’s interest to show the correct data. That’s the reason I think previous attempts to standardize the calculation of Days on Market failed, because how DOM is calculated should be at the sole discretion of the MLS.

The reason we are all working together is to enable sharing of curated data at scale with assurance of validity.

One. Hundred. Percent. And trust me, when we as a consumer are not showing the data correctly that agents can see in the MLS, we hear about it. We absolutely want to show the correct data.

Insidious – ha, great word choice! The company I work for is trying its best to faithfully represent accurate MLS data, so I hope we’re not being seen as insidious here. I wrote this proposal because we’ve devoted a lot of time to this problem, but we still run into real problems with accurately representing Days on Market. I see this proposal as one way to solve it.

But this comment, and others I’ve received since writing this proposal, are a good reminder that I need to first focus on communicating what the problem is that we’re seeing before I put forth a solution. That will be my next step: trying to better articulate the problem, and soliciting feedback on ways to fix it.

If a particular API they pull from is problematic then a correction of that endpoint is deserved

Unfortunately, we as consumers have no leverage in the choice of APIs we get our data from. If it comes down to working with providers to change the way they do Days on Market, are you willing to help out with a push from the MLS end?

[StevenMayhue]

I appreciate your pointed response which demonstrates a common belief in data accuracy and good stewardship.
We are both consumers of data. We are both limited to what the producer is sharing with us, and when. I believe the calculation of DOM/CDOM varies across our industry, but this is at the discretion of the MLS as outlined in their Rules & Regs. These are one manner of detecting listing manipulation. CDOM resets to zero after a measured time off market, and this is probably as varied as DOM calculation.
I doubt I will have a measurable effect, but I can definitely speak to my management regarding the issue when you can better articulate problem.
What is it that is specific to DOM/CDOM that is becoming a blocker/resource strain at scale?
If we can bring forward the problem, a solution may present itself.

[bryanburgers]

@StevenMayhue Here's my first crack and more thoroughly communicating the problem. Want to look it over and give me some feedback? #140 (comment)

[bryanburgers]

Zenlist DaysOnMarket case studies

I wrote up a bunch of case studies for how Zenlist deals with DaysOnMarket across various MLSes and provider types, and how this proposal would improve on what we already have.

It's a bit wordy¹ but it has nice diagrams²!

https://gist.github.com/bryanburgers/8a450ac2ec584e80c1226163ae234ee8

It's written from a very consumer-centric viewpoint because that's the experience I have.

As always, I'm curious to hear about viewpoints and case studies from other companies, or feedback on this document or the proposal.

Footnotes

1. Ugh, is anything I write for Transport not wordy? Sorry everyone. I try to keep it short, but this topic has a surprising amount of nuance. ↩

2. Well, in my opinion anyway. ↩

[bryanburgers]

The problem with Days on Market, a narrative

Everyone say hello to Connie 👋. Connie is new to the real estate industry, but is building out a brilliant new product that everyone is excited about.

Connie has a few core beliefs when it comes to her product. First, she wants to be a good friend to the MLSs and be a good steward of their data. Second, she wants to use RESO Standards to reduce her time to market. And third, she wants to show accurate and up-to-date listing data.

But mostly, Connie just wants her product to succeed.

Basic timestamp replication

Connie decides to retrieve data using RESO Web APIs and quickly comes to the conclusion that basic timestamp replication would be the best for her. For the three MLSs that she's starting with – OneMLS, TwoMLS, and ThreeMLS – she sets up a schedule to query for new updates to listings frequently throughout the day.

Connie is quite pleased with the way this works. All of the listings that she shows on her website come from her own database which makes her site super fast, but because any changes to the listings in the MLS make it into her database quickly, she's still being a good steward of the data.

She doesn't even have to modify or interpret the data at all; her website shows the address, the status, the list price, and a bunch of other information using Data Dictionary standards. Things are looking good.

😞 But things aren't good. A few days later, she receives a call from an agent telling her that her site is displaying the wrong Days on Market. How can that be? She's displaying exactly what she's receiving from the MLS! That approach works for addresses. That approach works for statuses. That approach works for list prices. What's going on here?

Ah yes. Connie realizes that unlike the rest of the values she receives in a listing payload, Days on Market isn't really a fixed value that can be cached in her database; it's more of a fluid value that is dependent on both data and rules that are internal to the MLS and – especially – dependent on the passage of time.

Always go to the source

Connie, knowing that the true source for Days on Market is the MLS, decides the only way to 100% faithfully represent the Days on Market is to always consult the true source.

She does a little digging with OneMLS's API and realizes that when she queries for the listing the agent told her about now, the data she gets back accurately represents Days on Market changes, but the ModificationTimestamp she uses for replication hasn't changed change. "Is that right?" she wonders as she re-reads the description of the ModificationTimestamp field. But she decides maybe the listing wasn't modified per se, only the passage of time has occurred. Whatever. Connie doesn't have time for philosophy; she has a budding business to run and a bug to fix!

Anyway, with the discovery that whenever she queries OneMLS's API it gives her the current value for Days on Market, Connie tweaks her website a bit. When a request to one of her listing pages comes in, all of the data except Days on Market comes from her database – still super fast – but the Days on Market widget sends off a separate request to the MLS to get the most up-to-date value.

The few listings she checks look accurate so she deploys the fix.

🤕 Thirty-seven minutes later she receives a call from OneMLS asking why her site is flooding their server with API requests. They're glad her site seems to be getting good traffic, but if she doesn't stop sending requests immediately, she'll be either rate limited or the MLS will be forced to cut off her access entirely.

Oof. Big, embarrassing mistake. Connie realizes that while she was trying to stay on the MLS's good side by being a good steward of the MLS's data, she ended up being on their bad side because she wasn't being a good steward of the MLS's resources.

So even though the MLS is the source of truth, Connie realizes that if her site is going to show listing data to a non-trivial number of visitors, her site really does need to cache MLS data in a local database.

Reconcile active listings once a day

Fortunately for Connie, the solution is right there in the name of the field. It's Days on Market. Not Minutes on Market. Not Hours on Market. Days on Market.

Connie only needs to go back to the MLS once a day to get an up-to-date Days on Market value for all the active listings she's showing.

"But wait," Connie thinks, "when do MLSs actually update their Days on Market?" Let's all give a hand for Connie because she's thinking ahead. So one night, she finds an active listing, stays up until midnight, and then every 5 minutes requeries the API for that listing. Midnight: not updated. 12:05: nope. 12:10: nada. 12:15: wow, still no? Tired but persistent, she keeps at it. Ah, there it is, between 1:10 and 1:15 the Days on Market updated – oh that's right, that MLS is in the next timezone West.

So Connie sets up three new daily schedules, based on what timezones she believes each MLS to be in, and sets them to run at 1am local time for each – to give each MLS ample time to make sure all of the listings are updated.

Displaying the wrong Days on Market value for about an hour a day makes Connie feel like she's violating her "be a good steward of the data" values, but she mollifies herself by thinking most people are sleeping at that time of night anyway. And she also doesn't like that she now needs a custom schedule for each MLS – that feels kinda non-standard – but if it gets the job done, great!

😬 Things are mostly good, until the same agent gives her another call. He has three active listings that now all look correct, but he has one listing that is under contract with the wrong Days on Market. He doesn't particularly care, but he thought Connie might care so he called to give her a heads up.

Ah, that's right. Connie only reconciled active listings. Other listings also probably change their Days on Market, too. Connie thinks that can't be too hard to fix.

Reconcile more statuses once a day

"What statuses do need to be reconciled?" Connie wonders. Looking through the existing standards doesn't yield much, just that every MLS has their own rules. And she's not surprised to find out that there's no place any of the rules are published.

But that lingering fear of misusing the MLSs' resources again lingers, so she's determined not to reconcile every listing. By this time, she has amassed information about a lot of closed listings, and surely they don't have incrementing Days on Market, do they? She hopes not.

Not wanting to pull another all-nighter, Connie decides to call her contacts at the three MLSs. While they can't divulge their full rules, they do tell her which statuses factor into the calculation.

Unfortunately for Connie, the statuses from OneMLS that she needs to use when reconciling and the statuses from TwoMLS that she needs to use when reconciling are different statuses. Connie once again writes more MLS-specific code into her reconciler code for each MLS she works with so that she is reconciling the correct statuses. And she quietly hopes the MLSs don't change their rules without telling her.

The next morning, she checks the under contract listing and it has the right Days on Market value, so her reconciliation code is working great! Time to schedule a long-overdue night out with some of her friends!

😮‍💨 When she wakes up the next morning, she has a voicemail from another agent. Hoping she didn't make any calls she doesn't remember, she listens. Ah good, no ill-advised calls, but this agent is saying that Days on Market for ThreeMLS has been wrong on her site for weeks – active listings, under contract listings, pending listings, pretty much all of them except closed listings.

What? Is her reconciler not working like she thought? But she checked the results from OneMLS and everything was looking good?

Not all APIs work the same way

Connie starts digging into the API for ThreeMLS more closely. Really closely. She looks at her code. She looks at her data. She looks at her logs.

And finally, she sees it. The API for ThreeMLS behaves differently from the APIs for OneMLS and TwoMLS. The API for ThreeMLS only updates the Days on Market field when there is an unrelated change that causes the ModificationTimestamp field to update.

Is that standards compliant? Connie has no idea. What Connie does know is that this is the API ThreeMLS has given her access to and she can't change it's behavior. What Connie does know is that she has customers that use this MLS and she can't leave them hanging. What Connie does know is that she needs to figure out a way to make this work.

Aside: the Rolodex analogy

Connie decides to call her friend Melissa at ThreeMLS. So Connie gets out her trusty Rolodex and finds Melissa's information.

Yes, apparently this story takes place in some sort of a time anomaly where RESO Web API standards exist but people still commonly use a Rolodex. Don't ask.

Preparing to chit-chat, Connie refreshes all of the info she knows about Melissa – the names of her kids, that she likes to play tennis, and— huh, Connie wrote down that she's been at ThreeMLS for 2 years. But when did she write that?

When Connie wrote that information makes a big difference. If she wrote it 3 years ago, that would mean Melissa has been at ThreeMLS for 2+3=5 years. But if Connie wrote it yesterday (apparently the RESO/Rolodex time anomaly also sometimes affects Connie's memory) then Melissa has been at ThreeMLS for only 2+0=2 years.

Connie has an epiphany. The next time she talks to Melissa, she's going to ask her how long she's been at ThreeMLS and she's going to write down both the answer she gets back and when she received that answer! With those two bits of information together, she can always be accurate when talking to her friend. And for Days on Market...

Combining DaysOnMarket and ModificationTimestamp

Connie doesn't even call Melissa. Friends can wait, she has a solution. She puts on a pot of coffee and starts working late into the evening.

For ThreeMLS, she has two bits of useful information she can use. The data doesn't tell her what the Days on Market value is, it tells her what the Days on Market value was at a particular time, and that particular time is the ModificationTimestamp.

If Days on Market was 7, and it was that as of 4 days ago, then Days on Market now is 7+4=11.

So she writes up some code to calculate the number of days between the ModificationTimestamp and now, and add that to the number of Days on Market. She checks it against a few of listings in ThreeMLS, and it looks like it's working.

Surely she has everything figured out now, right?

🤔 The next day, she receives another call from her agent friend. Before she answers, she immediately knows what she did wrong.

She applied this code to all of the MLSs she works with, but now both OneMLS and TwoMLS are over-counting because their Days on Market values are as of this morning, not as of their old ModificationTimestamp values.

For OneMLS and TwoMLS, she has data telling her what the Days on Market value was, but not an associated time for when it was that value; ModificationTimestamp is certainly not the right timestamp.

Calculate based on when the data is fetched

Well, that's a pretty simple problem to fix, Connie decides. Instead of using the ModificationTimestamp, she'll use the time at which she fetched the data.

That will work for both OneMLS and TwoMLS because she reconciles the data every day, giving her an up-to-date Days on Market and an up-to-date fetch date.

And it will work for ThreeMLS because the fetch date will always be very close to the ModificationTimestamp.

Connie has now sunk a lot of time into being as accurate as possible with Days on Market, and she is feeling pretty good about the system she has put in place – custom MLS code and all. Success.

😏 You know exactly where this is going, don't you reader? It was not success. There is yet another problem.

And that problem is intermediaries.

Intermediaries

Connie's business is booming and she decides to bring on a fourth MLS – FourMLS. She determines what statuses she would need to reconcile on. She throws that information away because she determines that FourMLS's API behaves like ThreeMLS's API so reconciliation isn't an option.

She also determines that the API she gets data from is not run by FourMLS, it's run by an intermediary. Apparently that's because FourMLS still uses RETS internally. Connie doesn't know what RETS is, but that information doesn't seem like it should matter anyway, so she files it away.

She rolls out FourMLS and gets a bunch of new customers. Things are going well.

😠 And then, one day, suddenly, all of the Days on Market calculations for FourMLS are wrong.

Aside: Rolodex revisited

Connie has no idea what's wrong. To figure it out, she realizes she needs to call somebody at FourMLS. But she also realizes she doesn't have a contact there.

So instead Connie calls Melissa at ThreeMLS again. After chitchatting (and making sure to update her information about how long she's been working at the ThreeMLS – 4 years!) Connie asks if Melissa has a contact at FourMLS.

Melissa does! Her old high school friend Melvin works there. Melissa looks at her own Rolodex (still a time anomaly) and starts spewing information at Connie: Melvin doesn't have any children but he does have two cats; he's big into knitting; he's been at FourMLS for 2 years.

"Wait a second," Connie interrupts, "you say you have written on your Rolodex that he's been at FourMLS for 2 years, but when did you write that?" Melissa doesn't know, and doesn't quite understand what Connie is getting at.

Again, epiphany. Connie realizes that the intermediary API that FourMLS uses sometimes updates listings because they have changed in FourMLS directly. And sometimes the API updates listings because of work done in the API itself, like updating a schema mapping.

But what the intermediary API doesn't do is record any value that Connie can use as the when for calculating Days on Market. Connie can track the fetch time when she fetched from the intermediary API, but if the intermediary doesn't track its own fetch time, Connie still can't be accurate.

The Final Chapter?

With this fresh knowledge and no idea how to solve this problem, Connie has her greatest realization yet.

She stops trying to stay in sync on Days on Market entirely and instead changes the label on her site from "Days on Market" to "Days on Connie's Site", using the number of days between when her site first saw the listing and now. While that doesn't match any of the MLSs' calculations, at least she can work on her next big feature.

---

Recap

• DaysOnMarket is different from other fields
  • Other fields are static through time until the MLS says there is an update
  • DaysOnMarket is fluid with respect to time
• Reconciliation alone isn't the answer
  • Some APIs don't allow it at all
  • DOM is wrong between the start of the day and when reconciliation occurs
  • Consumers have to guess when they should start reconciling
  • The spec does not include a way to communicate which statuses to reconcile
• Trying to calculate based on info available isn't the answer
  • For some APIS, ModificationTimestamp is the best base timestamp
  • For some APIs, using the time that the record was fetched is the base timestamp
  • For some APIs, neither fetch time or ModificationTimestamp yields the right answer

[StevenMayhue]

Bryan,

Thank you for breaking this down to pencils and erasers for us.

I now have a better understanding of the issue you are attempting to address.
This I can discuss internally and give you a better reply.

[SergioDelRioT4Bi]

Bryan, that was absolutely AWESOME! I think this makes everyone truly understand what the problems with DOM and CDOM are.

[bryanburgers]

Reconciliation and auto-increment

During the Transport Working Group discussion there was some talk about whether we can re-use the DaysOnMarket field for this proposal or whether we need three completely new fields.

Much of that discussion centered around the fact that for some MLSes, DaysOnMarket is "calculated on output" by the API server in order to allow consumers to "reconcile" – to query for DaysOnMarket every day to get the most up-to-date value.

Much of the focus of this proposal has been around not requiring reconciliation at all (especially since reconciliation is not always possible). However, I am convinced this proposal and reconciliation work together nicely.

For APIs that do support reconciliation by calculating on output, does it make sense for DaysOnMarketRelativeTimestamp to be calculated on output, too? This would give the best of both worlds for consumers that are using reconciliation: they get both the most up-to-date value when they reconcile, and they get extra information that lets them stay in-sync in the early mornings until they reconcile again. For consumers that only use DaysOnMarket, backward compatibility is maintained and there is an easy path to upgrade to DaysOnMarket + auto-increment.

@SergioDelRioT4Bi @grispin I think you two were heavily in the "lets not re-use DaysOnMarket" camp. I'm curious if calculating DaysOnMarketRelativeTimestamp on output addresses any of your concerns and if there are other business needs that would be better addressed by not re-using DaysOnMarket.

[SergioDelRioT4Bi]

I am certainly not dead-set against re-using the same field. From my perspective, it was really more about isolating this change and to come up with more appropriately named fields for the future so it may be more understandable that this field needs special care and handling. I think it works just as you have outlined here.

[grispin]

My reason for not wanting to reuse the DOM field is to have the new DOM model strictly follow the replication standards. The current DOM field is tainted with all sorts of special handling and rules (don't touch timestamps when changing, no entity events, etc.) that producers and consumers implicitly know. This coupled with the change in The change in behaviours on the field are going to take some time to implement so we are going to need a deprecation period where both exist while consumers migrate from one solution to the new one and provider have both implemented to allow for this migration window.

The new DOM needs to follow all of the record handling rules that we have in the specification to make the replication issue easier to solve. Changes to the DOM start value, Timestamp and YN fields must trigger modification timestamp updates, entity events, webhook subscriptions, etc. Without this requirement we are not fully addressing the pain point that generating this issue. The goal is a DOM that is easily replication from a provider to a consumer without the consumer having to constantly check the data is accurate.

Consistent replication was why I was arguing against the the dynamic Timestamp field. If I pull the record yesterday, today and tomorrow and the modification timestamp has not changed, then neither should the records data fields. The fact that the data fields can change without notice makes replication more complex and causes the call back behaviors we see today. The new DOM fields should fallow all the modification timestamp and entity event rules to make replication easier. For this reason, the DOM fields should only be touched as infrequently as possible to reduce modification events to keep replication overhead low. The DOM fields should really only need to be changed when one of the fields that affect DOM rules are touched.

[bryanburgers]

Well put. That makes sense. Thank you.

[darnjo]

Issue created. See #150.