Brazilian Transparency Portal Contracts and Procurements Crawled from the public API

This is a crawl of the Web API provided by the Brazilian Federal Government for open budget data under the transparency portal (http://www.portaltransparencia.gov.br).<br>The datasets included in this crawl are the procurement (licitações), contracts (contratos) and government organizations (Organizações SIAFI). An introduction to the APIs is provided (http://www.portaltransparencia.gov.br/api-de-dados) and there is a swagger documentation available at http://www.transparencia.gov.br/swagger-ui.html.<br>Two additional undocumented APIs were also crawled:<br>- (/criterios/contratos/fornecedor/autocomplete) Surrogate IDs from CNPJ (a fiscal organization identifier);- (/pessoa-juridica/{id}/participante-licitacao/resultado) Participation of contractors in procurements (contractors are identified by their surrogate ID, not their CNPJ).<br>These undocumented APIs where only crawled for contractors that had contracts with organization number 26246 (Federal University of Santa Catarina).<br><br>The crawl includes data up to January 31st, 2020. The aforementioned datasets are updated monthly. <br><br>Software used to perform this crawl can be found at https://bitbucket.org/alexishuf/compsac-2020-experiments. This crawl-all.sh script does the full crawl (this requires 4 hours or more). More details of the crawling procedures can be found in the EXPERIMENTS.md file.<br>