Enable push/pull of Excel files for large-scale farm data

[danyelshan]

Dear @arruda,

I would like to ask if it would be possible to pull and push Excel files to the farm calendar. We are working with sheep farms with more than 600 animals per parcel, so it is not feasible to tag and push each animal’s information manually.

Thank you very much

[arruda]

Hello @danyelshan 🙂
I believe it should be possible to have a simple method for processing these large batches of assets (e.g., animals, crops, etc) by uploading a CSV file if we are to use the Open Agri Common Semantic Model (OCSM) to describe the data.
For example:

nationalID	name	description	hasAgriParcel	sex	isCastrated	species	breed	birthdate	isMemberOfAnimalGroup.hasName	status
RS	Cow	this is a cow	urn:farmcalendar:Parcel:00000000-0000-0000-0000-000000000003	2	false	cow	spotted	2025-09-01T16:09:00Z	livestock	1
DE123	Bella	A healthy dairy cow	urn:farmcalendar:Parcel:00000000-0000-0000-0000-000000000005	2	false	cow	Holstein	2023-05-15T00:00:00Z	dairy_herd	1
...

I could then add a feature to the farm calendar UI (as a "load animals from CSV file" button) on the same page where you would add a single animal.

[arruda]

Here is an example of how the animals might be represented in FarmCalendar using the OCSM (OpenAgri Common Semantic Model).
First how a sheep might look using the fields in our JSON-LD structure for the FarmAnimals endpoint in Farmcalendar:

{
  "@context": [
    "https://w3id.org/ocsm/main-context.jsonld"
  ],
  "@graph": [
    {
      "@type": "Animal",
      "@id": "urn:farmcalendar:FarmAnimal:12393f52-5179-5db4-b5b7-6f1dbbda654a", // automatic unique ID generated by FarmCalendar
      "nationalID": "RPF21039X",
      "name": "Sheep RPF21039X",  //some nickname for the animal or just have it the same as the ID 
      "description": "Assaf dairy ewe in lot 2...etc...etc...or leave empty",
      "hasAgriParcel": {  // the farm parcel where this animal is  
        "@type": "Parcel",
        "@id": "urn:farmcalendar:Parcel:00000000-0000-0000-0000-000000000003" // eg. FarmCalendar ID for CIA: ES08000579784Z
      },
      "sex": 2,
      "isCastrated": false,
      "species": "sheep",
      "breed": "Assaf",
      "birthdate": "2023-09-11T14:10:19.892134Z",  // the birthdate of the animal
      "isMemberOfAnimalGroup": {
        "@id": "urn:farmcalendar:AnimalGroup:d5793f52-5179-5db4-b5b7-6f1dbbda2e81",
        "@type": "AnimalGroup",
        "hasName": "9784Y" //  if CEXP is the animal group identification, perhaps thats the lote colum?
      },
      "status": 1,
      "invalidatedAtTime": null,
      "dateCreated": "2025-09-11T14:10:19.892134Z",
      "dateModified": "2025-09-11T14:10:19.892150Z"
    }
  ]
}

Next, is the data that is currently missing in FarmCalendar. I suggest the creation of a generic "AnimalObservation" or "AnimalActivity". This could be used to create a calendar activity referencing the animal with the title "Lactation 3 - Health & Production Check" or something of the sort. This activity would also group different generic observations related to this specific health check, like the Milk production, Fat, Protein, SCC and Number of lactation.

Here is an example of how these records of the lactating activities might look like in the FarmCalendar api data model:

{
  "@context": [
    "https://w3id.org/ocsm/main-context.jsonld"
  ],
  "@graph": [
    // 1.  Generic AnimalActivity for this lactation cycle (groups together all the observation records for this calendar event)
    {
      "@type": "AnimalActivity",
      "@id": "urn:farmcalendar:AnimalActivity:9f2e4a1d-5c83-44a8-b9a1-a0c2c1d3e4f5",
      "activityType": {
        "@type": "FarmCalendarActivityType",
        "@id": "urn:farmcalendar:FarmCalendarActivityType:00000000-0000-0000-0000-000000000100" // ID for generic AnimalActivity
      },
      "title": "Lactation 3 - Health & Production Check",
      "details": "Comprehensive milk recording and health check for lactation cycle starting 2024-11-01.",
      "hasStartDatetime": "2024-11-01T12:00:00Z", // Calving date (FECHA DE PARTO)
      "hasEndDatetime": "2025-01-24T12:00:00Z", // Calving date + Días (84)
      "hasAgriParcel": {
        "@type": "Parcel",
        "@id": "urn:farmcalendar:Parcel:00000000-0000-0000-0000-000000000003"
      },
      "hasAnimal": {
        "@type": "Animal",
        "@id": "urn:farmcalendar:FarmAnimal:12393f52-5179-5db4-b5b7-6f1dbbda654a" // reference to the animal UUID in Farmcalendar
      },
      "responsibleAgent": null,
      "usesAgriculturalMachinery": null,
      "isPartOfActivity": null
    },
    // 2.  OBSERVATION: Lactation Number
    {
      "@type": "Observation",
      "@id": "urn:farmcalendar:Observation:000e4a1d-5c83-44a8-b9a1-a0c2c1d01",
      "activityType": {
        "@type": "FarmCalendarActivityType",
        "@id": "urn:farmcalendar:FarmCalendarActivityType:00000000-0000-0000-0000-000000000101" // ID for LactationNumberObservation
      },
      "title": "Lactation Number",
      "phenomenonTime": "2024-11-01T12:00:00Z",
      "hasEndDatetime": "2025-01-24T12:00:00Z",
      "observedProperty": "LactationNumber",
      "hasResult": {
        "@type": "QuantityValue",
        "hasValue": "3" // From column 'L'
      },
      "isPartOfActivity": {
        "@id": "urn:farmcalendar:AnimalActivity:9f2e4a1d-5c83-44a8-b9a1-a0c2c1d3e4f5",  // reference to the Generic AnimalActivity for this lactation cycle
        "@type": "AnimalActivity"
      }
    },
    // 3.  OBSERVATION: Milk Yield (Kg/Day)
    {
      "@type": "Observation",
      "@id": "urn:farmcalendar:Observation:000e4a1d-5c83-44a8-b9a1-a0c2c1d02",
      "activityType": {
        "@type": "FarmCalendarActivityType",
        "@id": "urn:farmcalendar:FarmCalendarActivityType:00000000-0000-0000-0000-000000000102" // ID for MilkYieldObservation
      },
      "title": "Milk Yield",
      "phenomenonTime": "2024-11-01T12:00:00Z",
      "hasEndDatetime": "2025-01-24T12:00:00Z",
      "observedProperty": "MilkYield",
      "hasResult": {
        "@type": "QuantityValue",
        "unit": "KGM",
        "hasValue": "8" // K.D. from LACTACION NATURAL group
      },
      "isPartOfActivity": {
        "@id": "urn:farmcalendar:AnimalActivity:9f2e4a1d-5c83-44a8-b9a1-a0c2c1d3e4f5",  // reference to the Generic AnimalActivity for this lactation cycle
        "@type": "AnimalActivity"
      }
    },
    // 4.  OBSERVATION: Fat Percentage
    {
      "@type": "Observation",
      "@id": "urn:farmcalendar:Observation:000e4a1d-5c83-44a8-b9a1-a0c2c1d03",
      "activityType": {
        "@type": "FarmCalendarActivityType",
        "@id": "urn:farmcalendar:FarmCalendarActivityType:00000000-0000-0000-0000-000000000103" // ID for FatPercentageObservation
      },
      "title": "Fat Percentage",
      "phenomenonTime": "2024-11-01T12:00:00Z",
      "hasEndDatetime": "2025-01-24T12:00:00Z",
      "observedProperty": "FatPercentage",
      "hasResult": {
        "@type": "QuantityValue",
        "unit": "P1",
        "hasValue": "5.77" // %G from VERIFICACION group (example value)
      },
      "isPartOfActivity": {
        "@id": "urn:farmcalendar:AnimalActivity:9f2e4a1d-5c83-44a8-b9a1-a0c2c1d3e4f5",  // reference to the Generic AnimalActivity for this lactation cycle
        "@type": "AnimalActivity"
      }
    },
    // 5.  OBSERVATION: Protein Percentage
    {
      "@type": "Observation",
      "@id": "urn:farmcalendar:Observation:000e4a1d-5c83-44a8-b9a1-a0c2c1d04",
      "activityType": {
        "@type": "FarmCalendarActivityType",
        "@id": "urn:farmcalendar:FarmCalendarActivityType:00000000-0000-0000-0000-000000000104" // ID for ProteinPercentageObservation
      },
      "title": "Protein Percentage",
      "phenomenonTime": "2024-11-01T12:00:00Z",
      "hasEndDatetime": "2025-01-24T12:00:00Z",
      "observedProperty": "ProteinPercentage",
      "hasResult": {
        "@type": "QuantityValue",
        "unit": "P1",
        "hasValue": "4.39" // %P from VERIFICACION group (example value)
      },
      "isPartOfActivity": {
        "@id": "urn:farmcalendar:AnimalActivity:9f2e4a1d-5c83-44a8-b9a1-a0c2c1d3e4f5",  // reference to the Generic AnimalActivity for this lactation cycle
        "@type": "AnimalActivity"
      }
    },
    // 6. OBSERVATION: Somatic Cell Count
    {
      "@type": "Observation",
      "@id": "urn:farmcalendar:Observation:000e4a1d-5c83-44a8-b9a1-a0c2c1d05",
      "activityType": {
        "@type": "FarmCalendarActivityType",
        "@id": "urn:farmcalendar:FarmCalendarActivityType:00000000-0000-0000-0000-000000000105" // ID for SomaticCellCountObservation
      },
      "title": "Somatic Cell Count",
      "phenomenonTime": "2024-11-01T12:00:00Z",
      "hasEndDatetime": "2025-01-24T12:00:00Z",
      "observedProperty": "SomaticCellCount",
      "hasResult": {
        "@type": "QuantityValue",
        "unit": "NMB",
        "hasValue": "633" // RCS from LACTACION NATURAL group
      },
      "isPartOfActivity": {
        "@id": "urn:farmcalendar:AnimalActivity:9f2e4a1d-5c83-44a8-b9a1-a0c2c1d3e4f5",  // reference to the Generic AnimalActivity for this lactation cycle
        "@type": "AnimalActivity"
      }
    }
  ]
}

For now, what is missing in this last part is only the generic AnimalActivity (that is, a built-in activity type with a reference to a specific Animal registered in FarmCalendar).

This data modeling would also benefit the management of the animal as somenting separated from it's lactating and milk production management. The animal information stays static, while it milk production becomes dynamic records in time which should also be helpfull when desining reports on the milk production of each animal over a given time period.

I'll check with @rapw3k on the fine details for data modeling for the generic animal activity. I'll create a issue for this in the OCSM repository to separete the data modeling decision, from this issue. Once we decide on the data model for this new animal activity, I'll update this issue with how it would look like.

[danyelshan]

Hello @arruda, thanks again for the detailed explanation — just to make sure we understand the intended workflow correctly.

For large sheep farms, our main concern is whether these fields would be handled automatically during CSV import, or if they would require manual creation of activities per animal.

Concretely, for the following fields:

Sex (“Sexo”)

Birth date (“Nacimiento”)

Milk production (Kg/day) – “K.D.”

Fat (%) – “%G”

Protein (%) – “%P”

Somatic Cell Count (SCC) – “RCS”

Number of lactation – “L”

Farm / Parcel ID – “CIA”

We assume that Sex, Birth date and CIA would map directly to the static Animal entity.

Our key question is about the production-related fields (K.D., %G, %P, RCS, L):

Would importing a CSV automatically create the corresponding AnimalActivity (e.g. lactation or milk control) and its associated Observations for each animal?
Or would users need to manually create an activity per animal to record these values?

For farms with hundreds of animals per parcel, the latter would unfortunately not be feasible, so automatic creation during import would be essential for adoption.

Thanks in advance for the clarification.

Best regards,
Daniel

[arruda]

Hello @danyelshan, these are good points you are raising. I'll discuss these and other finer details with the rest of the development team to ensure we move forward with a solution that addresses your issue and aligns with the project proposal. I'll update you once we get clarification on these concerns.

[danyelshan]

Thank you for the consideration. I appreciate you discussing this with the team. I’ll wait for your update once you have more clarity.

[arruda]

Hello @danyelshan, I've discussed this with the rest of the team, and what we will do is that we will also work on creating a script that will translate your CSV file into the necessary data format that you'll will need to push all this data animal at once to the farm calendar.

I still need to discuss with the rest of the dev team some fine details on the data model that will be used internally in the FC before implementing this batch uploading feature in FC, but once this is done I'll ping you here in this issue and will provide the step-by-step to use the new feature =)

[danyelshan]

Hi, thanks a lot for the update and for coordinating this with the rest of the team.

From our side, I’ll send you by the end of next week at the latest the definitive CSV file that we are currently using internally, so you can test it and move forward with the implementation and integration on your end.

Looking forward to your update once the internal data model details are settled. Thanks again!