Skip to content

codeforIATI/iati-data-dump

 
 

Repository files navigation

IATI Data Dump

A daily snapshot of all IATI data (and metadata) on the IATI registry.

Download all IATI data

Rationale

Lots of IATI products do a daily pull of all IATI XML data. That involves downloading gigabytes of data, via thousands of HTTP requests. So it’s quite slow.

Downloading a single archive file is significantly faster. That’s available here!

How does this work?

We perform a daily fetch of all IATI data, compress it, and upload the archive. There’s a metadata.json file inside the zip, that lets you know how fresh the data is.

This github gist is updated with the list of erroring datasets.

Acknowledgements

The original code relied heavily on IATI Registry Refresher, made by @caprenter and @Bjwebb.

The new, much faster code was contributed by @notshi and @xriss.

Contributing

Feel free to raise an issue or send a pull request.

About

📷 A daily snapshot of all IATI data on the IATI Registry

Topics

Resources

License

Code of conduct

Stars

Watchers

Forks

Languages

  • Shell 55.6%
  • Python 43.7%
  • Ruby 0.7%