Page MenuHomePhorge

Actually read in historical data from JSON response
Closed, ResolvedPublic

Assigned To
Authored By
keithzg
Sun, Jun 1, 3:52 PM
Tags
  • Restricted Project
  • Restricted Project
  • Restricted Project
  • Restricted Project
Referenced Files
None
Subscribers

Description

For some reason harder than it should be!

Revisions and Commits

Related Objects

StatusAssignedTask
Resolvedkeithzg

Event Timeline

keithzg triaged this task as High priority.

Okay so this is driving me insane.

PM 1 counts differ in structure between the historical and current readings like so:

  • historical
    • daily
      • some_day
        • pm1 is an integer value giving the concentration in µg/m³
    • hourly
      • some_hour
        • pm1 is an integer value giving the concentration in µg/m³
  • current
    • pm1 is an array
      • conc is an integer value giving the concentration in µg/m³

This isn't true of the PM2.5 and PM10 values, which are always arrays. So my logic, looking ahead to me eventually parsing historical data too to fill in gaps, went like so:

	# Initialize array (do we even have to do that in Python? I forget tbqh though I swear it's sometimes weird if ya don't)
	moment = {}
	
	# Get non-array data
	fields = ["ts", "tp", "hm", "pr"]
	for field in fields:
		logger.debug(data[field])
		moment[field] = data[field]
	
	# Get PM2.5 and PM10 values from their arrays
	fields = ["conc", "aqius", "aqicn"]
	
	for pm in ["25", "10"]:
		for field in fields:
			moment[f"pm{pm}_{field}"] = data[f"pm{pm}"][field]

	# PM1 is special, differs between historical and current
	if isinstance(data["pm1"], int):
		moment["pm1_conc"] = data["pm1"]
	else:
		for field in fields:
			moment[f"pm1_{field}"] = data["pm1"][field]
	
	return moment

But somehow, now actually parsing said historical data, I'm getting:

Traceback (most recent call last):
  File "/home/keithzg/Code/git/wxzg/test.py", line 147, in <module>
    main()
  File "/home/keithzg/Code/git/wxzg/test.py", line 135, in main
    write_to_db(data, args.include_past)
  File "/home/keithzg/Code/git/wxzg/test.py", line 47, in write_to_db
    current = parse_moment(i)
              ^^^^^^^^^^^^^^^
  File "/home/keithzg/Code/git/wxzg/test.py", line 105, in parse_moment
    moment[f"pm1_{field}"] = data["pm1"][field]
                             ~~~~~~~~~~~^^^^^^^
TypeError: 'float' object is not subscriptable

Which is like, wait, why is that happening? Oh it's a float? Surely it's effectively an int, and I would have thought isinstance() would work then, but lets check

	logging.debug("Checking type and value")
	logging.debug(type(data["pm1"]))
	logging.debug(data["pm1"])
	if isinstance(data["pm1"], int):
DEBUG:root:Checking type and value
DEBUG:root:<class 'float'>
DEBUG:root:2.2

Okay, fair, I really thought this value was only ever returning ints but I guess not! Okay so lets check for floats---

DEBUG:root:<class 'int'>
DEBUG:root:3
Traceback (most recent call last):
  File "/home/keithzg/Code/git/wxzg/test.py", line 147, in <module>
    main()
  File "/home/keithzg/Code/git/wxzg/test.py", line 135, in main
    write_to_db(data, args.include_past)
  File "/home/keithzg/Code/git/wxzg/test.py", line 47, in write_to_db
    current = parse_moment(i)
              ^^^^^^^^^^^^^^^
  File "/home/keithzg/Code/git/wxzg/test.py", line 105, in parse_moment
    moment[f"pm1_{field}"] = data["pm1"][field]
                             ~~~~~~~~~~~^^^^^^^
TypeError: 'int' object is not subscriptable

Autotyping strikes again, fuck I'm getting radicalized...

No, no, it's fine, #Python is fine, I'll just go

	if isinstance(data["pm1"], (int, float)):

and everything will be fine, just fine...

keithzg closed this task as Resolved by committing Restricted Diffusion Commit.Sun, Jun 1, 4:18 PM
keithzg added a commit: Restricted Diffusion Commit.
keithzg changed the visibility from "All Users" to "Public (No Login Required)".Sun, Jun 1, 4:18 PM