FetchData module¶

This module handles all operations involving fetching reports.

The process of fetching reports is made up of these steps:

Each vendor is queried for its supported reports using the SUSHI API
The vendor is then queried for each supported report, also using the SUSHI API
The raw JSON response is converted to JsonModel objects that make it easier to work with the JSON data.
The model objects are then used to create create row objects that will be in the final TSV file.
The row objects are then sorted by their primary columns, for example, item reports are sorted by the item column.
The sorted rows are then used to create and save a final TSV report file that adheres to the COUNTER 5 standards.
After all reports are processed, the database is updated with the new data

Note

All fetch operations are multi-threaded. Each vendor has it’s own thread, each report for that vendor also has it’s own thread. The maximum concurrent vendors and reports (per vendor) can be changed in the settings tab on the GUI

class FetchData.DatabaseReportItemModel(database, publisher, item_ids, publisher_ids, platform, data_type, access_method, performances)¶

Bases: GeneralUtils.JsonModel

Models a SUSHI Database Report Item

classmethod from_json(json_dict)¶

class FetchData.ExceptionModel(code, message, severity, data)¶

Bases: GeneralUtils.JsonModel

Models a SUSHI Exception

classmethod from_json(json_dict)¶

class FetchData.FetchReportsAbstract(vendors, settings, widget)¶

Bases: object

cancel_workers()¶: Sends a cancel signal to all vendor workers, updates the UI accordingly

fetch_vendor_data(request_data)¶

Initiates the process to fetch reports from a vendor

This creates a new thread to work on this vendor

Parameters: request_data (RequestData) – The request data for this vendor request

finish_fetching_reports()¶: Finishes up the fetch process

finish_updating_database()¶: Finishes up the database update process

get_result_widget(vendor, vendor_widget, process_result)¶

This creates a result widget for either a vendor or a vendor’s report

Parameters

vendor (Vendor) – The target vendor
vendor_widget (QWidget) – The vendor’s widget in the fetch progress dialog
process_result (ProcessResult) – The result to show

Return type

QWidget

is_yearly_range(begin_date, end_date)¶

Checks if a date range will retrieve all available reports for one year

Parameters

begin_date (QDate) – The begin date
end_date (QDate) – The end date

Return type

bool

on_report_to_retry_toggled(checked_state, vendor, report_type)¶

Handles the signal emmited when a report result’s retry checkbox is toggled

The report is added the the list of reports to be retried when the ‘retry selected reports’ button is clicked

Parameters

checked_state (int) – The new checked state
vendor (Vendor) – The vendor that this report belongs to
report_type – The report type of the report

on_vendor_worker_finished(worker_id)¶

Handles the signal emmited when a vendor worker has finished

Parameters: worker_id (str) – The worker ID of the vendor

on_vendors_changed(vendors)¶

Handles the signal emitted when the system’s vendor list is updated

Parameters: vendors (list) – An updated list of the system’s vendors

retry_selected_reports(progress_window_title)¶

Retries the selected reports to retry

Parameters: progress_window_title (str) – The title of the fetch progress dialog

start_progress_dialog(window_title)¶

Sets up and shows the fetch progress dialog

Parameters: window_title (str) – The title of the fetch progress dialog

start_updating_database()¶

Starts a thread to update the database. Returns True if successfully started

Return type: bool

update_results_ui(vendor, vendor_result=None, report_results=None)¶

Updates the fetch progress dialog to show results

Parameters

vendor (Vendor) – The vendor being updated
vendor_result (Optional[ProcessResult]) – The result of the vendor
report_results (Optional[list]) – The results of the vendor’s reports

update_vendors(vendors)¶

Updates the local copy of vendors that support report fetching (SUSHI)

Parameters: vendors (list) – A list of vendors

update_vendors_ui()¶: Updates the UI to show vendors that support report fetching (SUSHI)

class FetchData.FetchReportsController(vendors, settings, widget, fetch_reports_ui)¶

Bases: FetchData.FetchReportsAbstract

Controls the Fetch Reports tab

This class fetches master and standard reports with default parameters. The generated TSV files only include the mandatory columns for each report type

Parameters

vendors (list) – The list of vendors in the system
settings (SettingsModel) – The system’s user settings
widget (QWidget) – The fetch reports widget
fetch_reports_ui (Ui_fetch_reports_tab) – The UI for the fetch reports widget

custom_dir_frame_message_show(begin_date, end_date)¶

Checks which message will show on the custom dir frame

Parameters

begin_date (QDate) – The begin date
end_date (QDate) – The end date

Return type

bool

deselect_all_report_types()¶: Un-checks all report types in the report types list view

deselect_all_vendors()¶: Un-checks all vendors in the vendors list view

fetch_advanced_data()¶: Fetches reports based on the selected options in the advanced view of the UI

fetch_all_basic_data()¶: Fetches all reports for the selected year

on_custom_dir_clicked()¶: Handles the signal emitted when the choose custom directory button is clicked

on_date_month_changed(month, date_type)¶

Handles the signal emitted when a date’s month is changed

Parameters

month (int) – The new month
date_type (str) – The date to be updated

on_date_year_changed(date, date_type)¶

Handles the signal emitted when a date’s year is changed

Parameters

date (QDate) – The new date
date_type (str) – The date to be updated

on_fetch_all_date_changed(date)¶

Handles the signal emitted when the ‘fetch all’ date is changed

Parameters: date (QDate) – The new date

select_all_report_types()¶: Checks all report types in the report types list view

select_all_vendors()¶: Checks all vendors in the vendors list view

update_vendors_ui()¶: Updates the UI to show vendors that support report fetching (SUSHI)

class FetchData.FetchSpecialReportsController(vendors, settings, widget, fetch_special_reports_ui)¶

Bases: FetchData.FetchReportsAbstract

deselect_all_vendors()¶: Un-checks all vendors in the vendors list view

fetch_special_data()¶: Fetches reports based on the selected options in the UI

on_custom_dir_clicked()¶: Handles the signal emitted when the choose custom directory button is clicked

on_date_month_changed(month, date_type)¶

Handles the signal emitted when a date’s month is changed

Parameters

month (int) – The new month
date_type (str) – The date to be updated

on_date_year_changed(date, date_type)¶

Handles the signal emitted when a date’s year is changed

Parameters

date (QDate) – The new date
date_type (str) – The date to be updated

on_report_type_selected(major_report_type)¶

Handles the signal emitted when a report type is selected

Parameters: major_report_type (MajorReportType) – The major report type (Only master reports are supported)

on_special_date_parameter_option_button_clicked(special_option, line_edit)¶

Handles the signal emitted when a date special option is clicked

Parameters

special_option – The special option
line_edit – The line edit to show the selected date parameters for this option

on_special_option_toggled(option, is_checked)¶

Handles the signal emitted when a special option is checked or un-checked

Parameters

option (str) – The special option
is_checked (bool) – Checked or un-checked

on_special_parameter_option_button_clicked(special_option, line_edit)¶

Handles the signal emitted when a special option with parameters clicked

Parameters

special_option – The special option
line_edit – The line edit to show the selected parameters for this option

select_all_vendors()¶: Checks all vendors in the vendors list view

update_vendors_ui()¶: Updates the UI to show vendors that support report fetching (SUSHI)

class FetchData.InstanceModel(metric_type, count)¶

Bases: GeneralUtils.JsonModel

Models a SUSHI Instance

classmethod from_json(json_dict)¶

class FetchData.ItemComponentModel(item_name, item_ids, item_contributors, item_dates, item_attributes, data_type, performances)¶

Bases: GeneralUtils.JsonModel

Models a SUSHI Item Component

classmethod from_json(json_dict)¶

class FetchData.ItemContributorModel(item_type, name, identifier)¶

Bases: GeneralUtils.JsonModel

Models a SUSHI Item Contributor

classmethod from_json(json_dict)¶

class FetchData.ItemParentModel(item_name, item_ids, item_contributors, item_dates, item_attributes, data_type)¶

Bases: GeneralUtils.JsonModel

Models a SUSHI Item Parent

classmethod from_json(json_dict)¶

class FetchData.ItemReportItemModel(item, item_ids, item_contributors, item_dates, item_attributes, platform, publisher, publisher_ids, item_parent, item_components, data_type, yop, access_type, access_method, performances)¶

Bases: GeneralUtils.JsonModel

Models a SUSHI Item Report model

classmethod from_json(json_dict)¶

class FetchData.NameValueModel(name, value)¶

Bases: GeneralUtils.JsonModel

Models SUSHI models that are made up of Name and value

classmethod from_json(json_dict)¶

class FetchData.PerformanceModel(period, instances)¶

Bases: GeneralUtils.JsonModel

Models a SUSHI Performance

classmethod from_json(json_dict)¶

class FetchData.PeriodModel(begin_date, end_date)¶

Bases: GeneralUtils.JsonModel

Models a SUSHI Period

classmethod from_json(json_dict)¶

class FetchData.PlatformReportItemModel(platform, data_type, access_method, performances)¶

Bases: GeneralUtils.JsonModel

Models a SUSHI Platform Report Item

classmethod from_json(json_dict)¶

class FetchData.ProcessResult(vendor, report_type=None)¶

Bases: object

This holds the results of an fetch process

Parameters

vendor (Vendor) – The target vendor
report_type (Optional[str]) – The target report type

exception FetchData.ReportHeaderMissingException(exceptions)¶

Bases: Exception

An exception raised when a report header is missing from a report

class FetchData.ReportHeaderModel(report_name, report_id, release, institution_name, institution_ids, report_filters, report_attributes, exceptions, created, created_by)¶

Bases: GeneralUtils.JsonModel

Models a SUSHI Report Header

classmethod from_json(json_dict)¶

class FetchData.ReportModel(report_header, report_items)¶

Bases: GeneralUtils.JsonModel

Models a SUSHI Report

classmethod from_json(json_dict)¶

classmethod process_exceptions(json_dict)¶

Gets all exception models in a JSON dict, returns them as a list

Parameters

json_dict (dict) – A JSON dict

Raises

ReportHeaderMissingException – When the report header is missing
RetryLaterException – When a retry later exception model is received
UnacceptableCodeException – When the report cannot be processed based on the exception code

Return type

list

class FetchData.ReportRow(begin_date, end_date)¶

Bases: object

This models a row in the generated report, it contains every possible column

Parameters

begin_date (QDate) – The begin date of the request, used to populate the month columns in the report
end_date (QDate) – The end date of the request, used to populate the month columns in the report

class FetchData.ReportWorker(worker_id, report_type, request_data)¶

Bases: PyQt5.QtCore.QObject

This does all the work for a report

Parameters

worker_id (str) – The ID to identify this worker (vendor_name-report_type)
report_type (str) – The report type to be processed
request_data (RequestData) – The request data for this request

static add_report_header_to_file(report_header, file, include_attributes)¶

Adds the report header to a TSV file

Parameters

report_header (ReportHeaderModel) – The report header model
file – The TSV file to write to
include_attributes (bool) – Include the Report_Attributes value

static add_report_rows_to_file(report_type, report_rows, begin_date, end_date, file, include_all_attributes, is_special=False, special_options=None)¶

Adds the report’s rows to a TSV file

Parameters

report_type (str) – The report type
report_rows (list) – The report’s rows
begin_date (QDate) – The first date in the report
end_date (QDate) – The last date in the report
file – The TSV file to write to
include_all_attributes (bool) – Option to include all possible attributes for this report type to the report
is_special (bool) – If this is a special report
special_options (Optional[SpecialReportOptions]) – The special options if this is a special report

Return type

bool

make_request()¶: Sends the request fetch a report

notify_worker_finished()¶: Notifies any listeners that this worker has finished

process_report_model(report_model)¶

Processes the report model into a TSV report

Parameters: report_model (ReportModel) – The report model

process_response(response)¶

Processes the response from a request

Converts the receoved JSON to a SUSHI Report Model

Parameters: response (Response) – The received response

save_json_file(json_string)¶

Saves a raw JSON file of the report

Parameters: json_string (str) – The JSON string

save_tsv_files(report_header, report_rows)¶

Saves the TSV file in the target directories

Parameters

report_header – The SUSHI report header model
report_rows (list) – The report’s rows

static sort_rows(report_rows, major_report_type)¶

Sorts the rows of the report

Parameters

report_rows (list) – The report’s rows
major_report_type (MajorReportType) – The major report type of this report type

Return type

list

work()¶: Processes the report request

worker_finished_signal¶

class FetchData.RequestData(vendor, target_report_types, begin_date, end_date, save_location, settings, special_options=None)¶

Bases: object

This holds the data about a report request

Parameters

vendor (Vendor) – The vendor being processed
target_report_types (list) – The report types to attempt to fetch
begin_date (QDate) – The begin date to specify in the request
end_date (QDate) – The end date to specify in the request
save_location (str) – Where the generated report should be saved
settings (SettingsModel) – The system’s settings object
special_options (Optional[SpecialReportOptions]) – Special options if fetching a special report

exception FetchData.RetryLaterException(exceptions)¶

Bases: Exception

An exception raised when a retry later exception code is received in an exception model

class FetchData.SpecialReportOptions¶

Bases: object

This holds all the parameters that are used to process a special report

The options are stored as tuples, (option has non-default value, option type, option name, list of option values)

class FetchData.SupportedReportModel(report_id)¶

Bases: GeneralUtils.JsonModel

Models a SUSHI Supported Report

classmethod from_json(json_dict)¶

class FetchData.TitleReportItemModel(title, item_ids, platform, publisher, publisher_ids, data_type, section_type, yop, access_type, access_method, performances)¶

Bases: GeneralUtils.JsonModel

Models a SUSHI Title Report Item

classmethod from_json(json_dict)¶

class FetchData.TypeValueModel(item_type, value)¶

Bases: GeneralUtils.JsonModel

Models SUSHI models that are made up of Type and value

classmethod from_json(json_dict)¶

exception FetchData.UnacceptableCodeException(exceptions)¶

Bases: Exception

An exception raised when a report cannot be processed based on an exception code

class FetchData.VendorWorker(worker_id, request_data)¶

Bases: PyQt5.QtCore.QObject

This does all the work for a vendor when fetching reports

Parameters

worker_id (str) – The ID to identify this worker (vendor_name)
request_data (RequestData) – The request data for this request

check_for_exception(json_response)¶

Checks a JSON response for exception models

Parameters: json_response – The JSON response to be processed
Return type: list

fetch_report(report_type)¶

Initiates the process to fetch a report

Parameters: report_type (str) – The target report type

notify_worker_finished()¶: Notifies any listeners that this worker has finished

on_report_worker_finished(worker_id)¶

Handles the signal emmited when a report worker has finished

Parameters: worker_id (str) – The report worker’s worker id

process_response(response)¶

Processes the response from a REST request

Requests the target reports that are supported by the vendor

set_cancelling()¶: Sets the worker to a cancelling state

work()¶

Processes the vendor’s requests

Requests the vendor’s supported reports before requesting only the supported reports

worker_finished_signal¶

FetchData.exception_models_to_message(exceptions)¶

Formats a list of exception models into a single string

Return type: str

FetchData.get_models(model_key, model_type, json_dict)¶

This converts json lists into a list of the specified SUSHI model type

Parameters

model_key (str) – The target key to get the list of JSONObjects
model_type – The target model type, e.g PerformanceModel
json_dict (dict) – The JSON dict to get the list from

Return type

list

FetchData.get_month_years(begin_date, end_date)¶

Returns a list of month-year (MMM-yyyy) strings within a date range

Return type: list