Brazilian Transparency Portal Contracts and Procurements Crawled from the public API

This is a crawl of the Web API provided by the Brazilian Federal Government for open budget data under the transparency portal (http://www.portaltransparencia.gov.br).The datasets included in this crawl are the procurement (licitações), contracts (contratos) and government organizations (Organizações SIAFI). An introduction to the APIs is provided (http://www.portaltransparencia.gov.br/api-de-dados) and there is a swagger documentation available at http://www.transparencia.gov.br/swagger-ui.html.Two additional undocumented APIs were also crawled:- (/criterios/contratos/fornecedor/autocomplete) Surrogate IDs from CNPJ (a fiscal organization identifier);- (/pessoa-juridica/{id}/participante-licitacao/resultado) Participation of contractors in procurements (contractors are identified by their surrogate ID, not their CNPJ).These undocumented APIs where only crawled for contractors that had contracts with organization number 26246 (Federal University of Santa Catarina).The crawl includes data up to January 31st, 2020. The aforementioned datasets are updated monthly. Software used to perform this crawl can be found at https://bitbucket.org/alexishuf/compsac-2020-experiments. This crawl-all.sh script does the full crawl (this requires 4 hours or more). More details of the crawling procedures can be found in the EXPERIMENTS.md file.