Potential performance problem with IF

[dachafra]

Hi!

I'm running a set of queries where I'm having performance issues when IF is used.

The query is over the e-Procurement Ontology for calculating a set of transparency indicators, and it looks like this:

PREFIX epo: <http://data.europa.eu/a4g/ontology#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX legal: <https://www.w3.org/ns/legal#>
PREFIX dcterms: <http://purl.org/dc/terms#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX dc: <http://purl.org/dc/elements/1.1/>

SELECT DISTINCT ?countryID ?year (COUNT(DISTINCT ?lot) AS ?amountLots) (SUM(if(?bidders = 1, 1, 0)) AS ?numSingleBidders) WHERE {

        ?proc a epo:Procedure .
        ?proc epo:hasProcedureType ?p .
        FILTER ( ?p != <http://publications.europa.eu/resource/authority/procurement-procedure-type/neg-wo-call>)
        ?proc epo:hasProcurementScopeDividedIntoLot ?lot .

        ?stat a epo:SubmissionStatisticalInformation .
        ?stat epo:concernsSubmissionsForLot ?lot .
        ?stat epo:hasReceivedTenders ?bidders .

        ?resultnotice a epo:ResultNotice .
        ?resultnotice epo:refersToProcedure ?proc .
        ?resultnotice epo:hasDispatchDate ?ddate .
        BIND(year(xsd:dateTime(?ddate)) AS ?year) .

        ?resultnotice epo:refersToRole ?buyerrole .

        {
          SELECT DISTINCT ?buyerrole ?countryID WHERE {
            ?buyerrole epo:playedBy ?org .
            ?org legal:registeredAddress ?orgaddress .
            ?orgaddress epo:hasCountryCode ?countrycode  .
            ?countrycode dc:identifier ?countryID .

            ?org epo:hasBuyerType ?buytype .
            FILTER (?buytype != <http://publications.europa.eu/resource/authority/buyer-legal-type/eu-int-org> )
            }
          }
        } GROUP BY ?countryID ?year

If I remove from the projection (SUM(if(?bidders = 1, 1, 0)) AS ?numSingleBidders) the query works perfectly and runs in a reasonable time. At the moment that I add the aggregation, it's not able to run it in less than 5 minutes.

FYI: the same query over the same dataset takes around 0.31 seconds in Virtuoso (7.2.5).

Thanks!

[D063520]

Hi!

Honestly I do not know this piece of SPARQL syntax. If I understand your query there is a easy way to rewrite it. Instead of:

?stat epo:hasReceivedTenders ?bidders .

you can put

?stat epo:hasReceivedTenders ?bidders .
FILTER (?bidders = 1)

and eliminate the IF.

To debug further we would need the unterlying data. Can you share it?

[D063520]

PS: I would be curious to see the dataset and the workload if it is public. And also the comparison that your are doing if you can share at some point ....

[dachafra]

I'll check if we can provide you with the data (it's actually open data, but I need to check).

Yep, the results will be openly available soon.

[dachafra]

Thanks for the reply. I have changed to your proposal and the result is still the same (5 minutes time out)

PREFIX epo: <http://data.europa.eu/a4g/ontology#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX legal: <https://www.w3.org/ns/legal#>
PREFIX dcterms: <http://purl.org/dc/terms#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX dc: <http://purl.org/dc/elements/1.1/>
SELECT DISTINCT ?countryID ?year (COUNT(DISTINCT ?lot) AS ?amountLots) (SUM(?bidders) AS ?numSingleBidders) WHERE {

        ?proc a epo:Procedure .
        ?proc epo:hasProcedureType ?p .
        FILTER ( ?p != <http://publications.europa.eu/resource/authority/procurement-procedure-type/neg-wo-call>)
        ?proc epo:hasProcurementScopeDividedIntoLot ?lot .

        ?stat a epo:SubmissionStatisticalInformation .
        ?stat epo:concernsSubmissionsForLot ?lot .
        ?stat epo:hasReceivedTenders ?bidders .
        FILTER(?bidder = 1)

        ?resultnotice a epo:ResultNotice .
        ?resultnotice epo:refersToProcedure ?proc .
        ?resultnotice epo:hasDispatchDate ?ddate .
        BIND(year(xsd:dateTime(?ddate)) AS ?year) .

        ?resultnotice epo:refersToRole ?buyerrole .

        {
          SELECT DISTINCT ?buyerrole ?countryID WHERE {
            ?buyerrole epo:playedBy ?org .
            ?org legal:registeredAddress ?orgaddress .
            ?orgaddress epo:hasCountryCode ?countrycode  .
            ?countrycode dc:identifier ?countryID .

            ?org epo:hasBuyerType ?buytype .
            FILTER (?buytype != <http://publications.europa.eu/resource/authority/buyer-legal-type/eu-int-org> )
            }
          }
        } GROUP BY ?countryID ?year

In any case, I'll upload the data to a public place so you can use it for testing

[D063520]

Another idea would be that you put:

?stat epo:hasReceivedTenders "1"^^xsd:integer .

Not sure about the xsd integer (depend on the datatype) and you do:

(SUM(?stat) AS ?numSingleBidders)

Should give the same result.

[dachafra]

Same problem... Here it's the dataset I'm currently using: https://delicias.dia.fi.upm.es/nextcloud/index.php/s/X5foCwyLHYYKpYe

load: curl "http://127.0.0.1:1234/api/endpoint/load" -F "file=@2018_complete.nt"
run: curl -H 'Accept: text/csv' localhost:1234/api/endpoint/sparql --data-urlencode 'query@query.rq'

[dachafra]

any update on this?