Metric Service

SQS Que

The Metric Service listend the following AWS SQS queue:

• sqs.queue.url = https://sqs.us-east-1.amazonaws.com/380757016349/of-metrics-dev-processMetrics-queue
• sqs.queue.arn = arn:aws:sqs:us-east-1:380757016349:of-metrics-dev-processMetrics-queue

Development

Requirements

• JDK 11

Inputs

The Metric Service expects a subset of these data elements:

[
  {
    "batchId": "700123",
    "batchName": "CarrierClaimLoad",
    "fileName": "OPTIONAL",
    "fileReceivedDateTime": "OPTIONAL",
    "startDateTime": "2022-01-01T08:00:00.123456",
    "endDateTime": "OPTIONAL",
    "rowsSubmitted": 500,
    "rowsProcessed": 480,
    "rowsFailed": 20,
    "status": "PROCESSED|IN_PROGRESS|FAILED",
    "phase": "OPTIONAL/PARSE|LOAD|POST",
    "errors": "OPTIONAL/CAN BE MULTIPLE"
  }
]

Status must be one of three values: PROCESSED, IN_PROGRESS or FAILED
Phase is an optional field that if included must be one of three values: PARSE, LOADor POST
Errors is an optional field that is included should be structured as such:

"errors": [
        {
            "errorCount": 5,
            "description": "Lorem ipsum dolor sit amet"
        },
        {
            "errorCount": 2,
            "description": "consectetur adipiscing elit"
        }
    ]

Process Start

When a service begins processing a file, it should notify the Metric Service by posting a message to the queue.

Sample File

[
  {
    "batchId": "700123",
    "batchName": "CarrierClaimLoad",
    "fileName": "carrier001.txt",
    "fileReceivedDateTime": "2022-01-01T08:00:00.123456",
    "startDateTime": "2022-01-01T08:01:00.123456",
    "rowsSubmitted": 500,
    "status": "IN_PROGRESS",
    "phase": "PARSE"
  }
]

Processing Updates

Status updates during processing would include more information

Sample File

[
  {
    "batchId": "700123",
    "batchName": "CarrierClaimLoad",
    "fileName": "carrier001.txt",
    "fileReceivedDateTime": "2022-01-01T08:00:00.123456",
    "startDateTime": "2022-01-01T08:01:00.123456",
    "rowsSubmitted": 500,
    "rowsProcessed": 250,
    "rowsFailed": 0,
    "status": "IN_PROGRESS",
    "phase": "PARSE"
  }
]

Processing Complete

After a service finishes processing a file, it should send a final status update.

Sample File

[
  {
    "batchId": "700123",
    "batchName": "CarrierClaimLoad",
    "fileName": "carrier001.txt",
    "fileReceivedDateTime": "2022-01-01T08:00:00.123456",
    "startDateTime": "2022-01-01T08:01:00.123456",
    "endDateTime": "2022-01-01T08:05:00.123456",
    "rowsSubmitted": 500,
    "rowsProcessed": 480,
    "rowsFailed": 20,
    "status": "PROCESSED",
    "phase": "PARSE",
    "errors": [
        {
            "errorCount": 12,
            "description": "Lorem ipsum dolor sit amet"
        },
        {
            "errorCount": 8,
            "description": "consectetur adipiscing elit"
        }
    ]
]

Persisting Inputs

The Metric Service will insert or update the database according to these rules:

• Insert: if composite primary key (batchId & batchName fields) not found
• Update: if composite primary key (batchId & batchName fields) found

Querying

None of the input parameters are required, but the request must include at least one parameter from the set of {fileName|batchId|startDate|endDate}.

Input Parameter: fileName

• Maps directly to the database field of the same name

Input Parameter: batchId

• Maps directly to the database field of the same name

Input Parameter: startDate

• ISO Date Format yyyy-MM-dd — for example, "2000-10-31".
• Maps directly to the database field of the same name

Input Parameter: endDate

• ISO Date Format yyyy-MM-dd — for example, "2000-10-31".
• Maps directly to the database field of the same name

Input parameter: errorLimit

• Does not map to any database field
• Specifies the number of unique error descriptions returned
• Defaults to the configurable maximum

Sample Payloads

Simple Metrics

Simple Metric search by batchId (single result)

http://localhost:8080/metrics/v1/simple?batchid=A177721

Response

[
  {
    "batchName": "DataImport-ABY",
    "rowsSubmitted": 200,
    "rowsProcessed": 193,
    "rowsFailed": 7,
    "phase": "PARSE",
    "batchId": "A177721",
    "fileName": "data-load-08152022.txt",
    "fileReceivedDateTime": "2022-08-15T08:00:00.123456",
    "startDateTime": "2022-08-15T08:01:00.123456",
    "endDateTime": "2022-08-15T08:05:00.123456",
    "executionTimeInSeconds": 240,
    "status": "PROCESSED"
  }
]

Simple Metric search by batchId (multiple result)

http://localhost:8080/metrics/v1/simple?batchid=CC20220701

Response

[
  {
    "batchName": "CarrierClaimLoad",
    "rowsSubmitted": 500,
    "rowsProcessed": 500,
    "rowsFailed": 0,
    "phase": "LOAD",
    "batchId": "CC20220701",
    "fileName": "CC-20220701.txt",
    "fileReceivedDateTime": "2022-07-01T08:00:00.123456",
    "startDateTime": "2022-07-01T08:01:00.123456",
    "endDateTime": "2022-07-01T08:05:00.123456",
    "executionTimeInSeconds": 240,
    "status": "PROCESSED"
  },
  {
    "batchName": "CarrierClaimParse",
    "rowsSubmitted": 500,
    "rowsProcessed": 500,
    "rowsFailed": 0,
    "phase": "PARSE",
    "batchId": "CC20220701",
    "fileName": "CC-20220701.txt",
    "fileReceivedDateTime": "2022-07-01T08:00:00.123456",
    "startDateTime": "2022-07-01T08:10:00.123456",
    "endDateTime": "2022-07-01T08:12:30.123456",
    "executionTimeInSeconds": 150,
    "status": "PROCESSED"
  },
  {
    "batchName": "CarrierClaimPost",
    "rowsSubmitted": 500,
    "rowsProcessed": 400,
    "rowsFailed": 100,
    "phase": "POST",
    "batchId": "CC20220701",
    "fileName": "CC-20220701.txt",
    "fileReceivedDateTime": "2022-07-01T08:00:00.123456",
    "startDateTime": "2022-07-01T08:13:00.123456",
    "endDateTime": "2022-07-01T08:17:30.123456",
    "executionTimeInSeconds": 270,
    "status": "PROCESSED"
  }
]

Simple Metric search by fileName (single result)

http://localhost:8080/metrics/v1/simple?filename=data-load-08152022.txt

Response

[
  {
    "batchName": "DataImport-ABY",
    "rowsSubmitted": 200,
    "rowsProcessed": 193,
    "rowsFailed": 7,
    "phase": "PARSE",
    "batchId": "A177721",
    "fileName": "data-load-08152022.txt",
    "fileReceivedDateTime": "2022-08-15T08:00:00.123456",
    "startDateTime": "2022-08-15T08:01:00.123456",
    "endDateTime": "2022-08-15T08:05:00.123456",
    "executionTimeInSeconds": 240,
    "status": "PROCESSED"
  }
]

Simple Metric search by startDate

Querying by start date only will return everything from that date and newer, assuming the start date is an acceptable number of day in the past.

http://localhost:8080/metrics/v1/simple?startdate=2022-08-17

Response

[
  {
    "batchName": "DataImport-ORY",
    "rowsSubmitted": 700,
    "rowsProcessed": 700,
    "rowsFailed": 0,
    "phase": "PARSE",
    "batchId": "A370641",
    "fileName": "data-load-08172022.txt",
    "fileReceivedDateTime": "2022-08-17T08:00:00.123456",
    "startDateTime": "2022-08-17T08:01:00.123456",
    "endDateTime": "2022-08-17T08:05:00.123456",
    "executionTimeInSeconds": 240,
    "status": "PROCESSED"
  }
]

Simple Metric search by startDate & endDate

http://localhost:8080/metrics/v1/simple?startdate=2022-08-02&enddate=2022-08-07

Response

[
  {
    "batchName": "CreateDevenir",
    "rowsSubmitted": 350,
    "rowsProcessed": 350,
    "rowsFailed": 0,
    "phase": "LOAD",
    "batchId": "D4887690",
    "fileName": "Devenir-Aug-05-2022.txt",
    "fileReceivedDateTime": "2022-08-05T11:00:00.123456",
    "startDateTime": "2022-08-05T15:11:00.234567",
    "endDateTime": "2022-08-05T15:11:05.345678",
    "executionTimeInSeconds": 5,
    "status": "PROCESSED"
  },
  {
    "batchName": "OutboundConfig",
    "rowsSubmitted": 23,
    "rowsProcessed": 0,
    "rowsFailed": 23,
    "phase": "LOAD",
    "batchId": "OC-123-4-ME",
    "fileName": "123.txt",
    "fileReceivedDateTime": "2022-08-04T09:00:00.123456",
    "startDateTime": "2022-08-04T15:09:00.234567",
    "endDateTime": "2022-08-04T09:11:05.345678",
    "executionTimeInSeconds": -21475,
    "status": "FAILED"
  }
]

Detailed Metrics

Detailed Metric search by filename

http://localhost:8080/metrics/v1/detailed?filename=CC-20220701.txt

Response

[
  {
    "batchId": "CC20220701",
    "fileName": "CC-20220701.txt",
    "fileReceivedDateTime": "2022-07-01T08:00:00.123456",
    "startDateTime": "2022-07-01T08:01:00.123456",
    "endDateTime": "2022-07-01T08:17:30.123456",
    "executionTimeInSeconds": 990,
    "status": "PROCESSED",
    "phases": [
      {
        "batchName": "CarrierClaimLoad",
        "phase": "LOAD",
        "status": "PROCESSED",
        "startDateTime": "2022-07-01T08:01:00.123456",
        "endDateTime": "2022-07-01T08:05:00.123456",
        "executionTimeInSeconds": 240,
        "rowsSubmitted": 500,
        "rowsProcessed": 500,
        "rowsFailed": 0,
        "errors": []
      },
      {
        "batchName": "CarrierClaimParse",
        "phase": "PARSE",
        "status": "PROCESSED",
        "startDateTime": "2022-07-01T08:10:00.123456",
        "endDateTime": "2022-07-01T08:12:30.123456",
        "executionTimeInSeconds": 150,
        "rowsSubmitted": 500,
        "rowsProcessed": 500,
        "rowsFailed": 0,
        "errors": []
      },
      {
        "batchName": "CarrierClaimPost",
        "phase": "POST",
        "status": "PROCESSED",
        "startDateTime": "2022-07-01T08:13:00.123456",
        "endDateTime": "2022-07-01T08:17:30.123456",
        "executionTimeInSeconds": 270,
        "rowsSubmitted": 500,
        "rowsProcessed": 400,
        "rowsFailed": 100,
        "errors": [
          {
            "errorCount": 60,
            "description": "Lorem ipsum dolor sit amet"
          },
          {
            "errorCount": 25,
            "description": "consectetur adipiscing elit"
          },
          {
            "errorCount": 10,
            "description": "sed do eiusmod tempor "
          },
          {
            "errorCount": 3,
            "description": "Ut enim ad minim veniam"
          },
          {
            "errorCount": 2,
            "description": "quis nostrud exercitation "
          }
        ]
      }
    ]
  }
]