Skip to content

Instantly share code, notes, and snippets.

@mtauraso

mtauraso/Reproduce Bug.ipynb

Created November 26, 2024 18:42
Show Gist options
  • Save mtauraso/6666f1fb5c6d9897779ed5c3563bf179 to your computer and use it in GitHub Desktop.
Save mtauraso/6666f1fb5c6d9897779ed5c3563bf179 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
Reproduce Bug.ipynb
{
"cells": [
{
"cell_type": "markdown",
"id": "80c1d9ea-6f7e-473b-8e6e-95d438668977",
"metadata": {},
"source": [
"# Construct a butler URL for request\n",
"This goes through steps from `DP02_13a_Image_Cutout_SciDemo.ipynb` needed to create a butler URL for a known object in DP0.2 data"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "c66adba6-73b9-4068-9501-74cf990352c1",
"metadata": {
"execution": {
"iopub.execute_input": "2024-11-26T18:17:39.140126Z",
"iopub.status.busy": "2024-11-26T18:17:39.140013Z",
"iopub.status.idle": "2024-11-26T18:17:40.317975Z",
"shell.execute_reply": "2024-11-26T18:17:40.317577Z",
"shell.execute_reply.started": "2024-11-26T18:17:39.140113Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Job phase is COMPLETED\n"
]
},
{
"data": {
"text/html": [
"<div><i>Table length=1</i>\n",
"<table id=\"table140139149412304\" class=\"table-striped table-bordered table-condensed\">\n",
"<thead><tr><th>lsst_patch</th><th>lsst_tract</th><th>s_region</th><th>access_format</th><th>access_url</th><th>dataproduct_subtype</th></tr></thead>\n",
"<thead><tr><th>int64</th><th>int64</th><th>str512</th><th>str128</th><th>object</th><th>str64</th></tr></thead>\n",
"<tr><td>17</td><td>4431</td><td>POLYGON ICRS 55.514085 -32.322253 55.790197 -32.322253 55.789845 -32.088924 55.514437 -32.088924</td><td>application/x-votable+xml;content=datalink</td><td>https://usdf-rsp.slac.stanford.edu/api/datalink/links?ID=butler%3A//dp02/20d28216-534a-4102-b8a7-1c7f32a9b78c</td><td>lsst.deepCoadd_calexp</td></tr>\n",
"</table></div>"
],
"text/plain": [
"<Table length=1>\n",
"lsst_patch lsst_tract ... dataproduct_subtype \n",
" int64 int64 ... str64 \n",
"---------- ---------- ... ---------------------\n",
" 17 4431 ... lsst.deepCoadd_calexp"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import lsst.geom as geom\n",
"from lsst.rsp import get_tap_service\n",
"from astropy import units as u\n",
"from astropy.coordinates import SkyCoord, Angle\n",
"import pyvo\n",
"from pyvo.dal.adhoc import DatalinkResults\n",
"\n",
"service = get_tap_service(\"tap\")\n",
"ra = 55.7467\n",
"dec = -32.2862\n",
"coord = SkyCoord(ra=ra*u.degree, dec=dec*u.degree, frame='icrs')\n",
"radius = .1 * u.deg\n",
"\n",
"spherePoint = geom.SpherePoint(ra*geom.degrees, dec*geom.degrees)\n",
"\n",
"query = \"SELECT lsst_patch, lsst_tract, s_region, \" + \\\n",
" \"access_format, access_url, dataproduct_subtype \" + \\\n",
" \"FROM ivoa.ObsCore \" + \\\n",
" \"WHERE dataproduct_type = 'image' \" + \\\n",
" \"AND obs_collection = 'LSST.DP02' \" + \\\n",
" \"AND dataproduct_subtype = 'lsst.deepCoadd_calexp' \" + \\\n",
" \"AND lsst_band = 'i' \" + \\\n",
" \"AND CONTAINS(POINT('ICRS', \" + str(coord.ra.value) + \\\n",
" \", \" + str(coord.dec.value) + \"), \" + \\\n",
" \"s_region) = 1\"\n",
"job = service.submit_job(query)\n",
"job.run()\n",
"job.wait(phases=['COMPLETED', 'ERROR'])\n",
"print('Job phase is', job.phase)\n",
"assert job.phase == 'COMPLETED'\n",
"\n",
"results = job.fetch_result()\n",
"\n",
"results.to_table()\n"
]
},
{
"cell_type": "markdown",
"id": "aae1d27e-8fe3-4776-917e-9222ff0fb58e",
"metadata": {},
"source": [
"# Reproduce the bug\n",
"This is the next thing someone would run on the way to generating a cutout. It crashes with an unhelpful stack trace involving xml parsing. It shouldn't be xml parsing as we will see in a moment.\n",
"\n",
"First we set up the request we will be making."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "4d1d9fbd-b93d-4e90-a4dc-d3c3dc774b81",
"metadata": {
"execution": {
"iopub.execute_input": "2024-11-26T18:17:40.320724Z",
"iopub.status.busy": "2024-11-26T18:17:40.320599Z",
"iopub.status.idle": "2024-11-26T18:17:40.323394Z",
"shell.execute_reply": "2024-11-26T18:17:40.323022Z",
"shell.execute_reply.started": "2024-11-26T18:17:40.320711Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Data link url is: https://usdf-rsp.slac.stanford.edu/api/datalink/links?ID=butler%3A//dp02/20d28216-534a-4102-b8a7-1c7f32a9b78c\n",
"Auth Session is: Support for dict_keys(['anonymous', 'lsst-token'])\n",
"Full match:https://usdf-rsp.slac.stanford.edu/api/tap/capabilities:{'anonymous'}\n",
"Full match:https://usdf-rsp.slac.stanford.edu/api/tap/availability:{'anonymous'}\n",
"Full match:https://usdf-rsp.slac.stanford.edu/api/tap/logControl:{'ivo://ivoa.net/sso#cookie', 'ivo://ivoa.net/sso#OpenID', 'ivo://ivoa.net/sso#saml2.0', 'ivo://ivoa.net/sso#BasicAA', 'ivo://ivoa.net/sso#OAuth'}\n",
"Base match:https://usdf-rsp.slac.stanford.edu/api/obstap/tables:{'lsst-token'}\n",
"Base match:https://usdf-rsp.slac.stanford.edu/api/ssotap/tables:{'lsst-token'}\n",
"Base match:https://usdf-rsp.slac.stanford.edu/api/obstap/async:{'lsst-token'}\n",
"Base match:https://usdf-rsp.slac.stanford.edu/api/ssotap/async:{'lsst-token'}\n",
"Base match:https://usdf-rsp.slac.stanford.edu/api/siav2/query:{'lsst-token'}\n",
"Base match:https://usdf-rsp.slac.stanford.edu/api/obstap/sync:{'lsst-token'}\n",
"Base match:https://usdf-rsp.slac.stanford.edu/api/ssotap/sync:{'lsst-token'}\n",
"Base match:https://usdf-rsp.slac.stanford.edu/api/tap/tables:{'ivo://ivoa.net/sso#cookie', 'ivo://ivoa.net/sso#OpenID', 'lsst-token', 'ivo://ivoa.net/sso#BasicAA', 'ivo://ivoa.net/sso#saml2.0', 'ivo://ivoa.net/sso#OAuth'}\n",
"Base match:https://usdf-rsp.slac.stanford.edu/api/tap/async:{'lsst-token'}\n",
"Base match:https://usdf-rsp.slac.stanford.edu/api/datalink:{'lsst-token'}\n",
"Base match:https://usdf-rsp.slac.stanford.edu/api/tap/sync:{'lsst-token'}\n",
"Base match:https://usdf-rsp.slac.stanford.edu/api/cutout:{'lsst-token'}\n",
"Base match:https://usdf-rsp.slac.stanford.edu/api/obstap:{'lsst-token'}\n",
"Base match:https://usdf-rsp.slac.stanford.edu/api/ssotap:{'lsst-token'}\n",
"Base match:https://usdf-rsp.slac.stanford.edu/api/siav2:{'lsst-token'}\n",
"Base match:https://usdf-rsp.slac.stanford.edu/api/tap:{'ivo://ivoa.net/sso#cookie', 'ivo://ivoa.net/sso#OpenID', 'lsst-token', 'ivo://ivoa.net/sso#BasicAA', 'ivo://ivoa.net/sso#saml2.0', 'ivo://ivoa.net/sso#OAuth'}\n"
]
}
],
"source": [
"dataLinkUrl = results[0].getdataurl()\n",
"auth_session = service._session\n",
"print(f\"Data link url is: {dataLinkUrl}\")\n",
"print(f\"Auth Session is: {auth_session}\")"
]
},
{
"cell_type": "markdown",
"id": "77bf5d97-74c1-4f58-89b6-934321af27f6",
"metadata": {},
"source": [
"# Error occurs here\n",
"When we use `DataLinkResults` to retrieve a url for this object for use by IVOA tools we get a crash."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "be8db8a2-1114-44cc-abf2-470f4b5b2767",
"metadata": {
"execution": {
"iopub.execute_input": "2024-11-26T18:17:40.323952Z",
"iopub.status.busy": "2024-11-26T18:17:40.323829Z",
"iopub.status.idle": "2024-11-26T18:17:40.631763Z",
"shell.execute_reply": "2024-11-26T18:17:40.631225Z",
"shell.execute_reply.started": "2024-11-26T18:17:40.323940Z"
}
},
"outputs": [
{
"ename": "ValueError",
"evalue": "1:0: not well-formed (invalid token)",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[3], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m dl_results \u001b[38;5;241m=\u001b[39m \u001b[43mDatalinkResults\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfrom_result_url\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdataLinkUrl\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2\u001b[0m \u001b[43m \u001b[49m\u001b[43msession\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mauth_session\u001b[49m\u001b[43m)\u001b[49m\n",
"File \u001b[0;32m/opt/lsst/software/stack/conda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/pyvo/dal/query.py:286\u001b[0m, in \u001b[0;36mDALResults.from_result_url\u001b[0;34m(cls, result_url, session)\u001b[0m\n\u001b[1;32m 279\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m 280\u001b[0m \u001b[38;5;124;03mCreate a result object from a url.\u001b[39;00m\n\u001b[1;32m 281\u001b[0m \n\u001b[1;32m 282\u001b[0m \u001b[38;5;124;03mUses the optional session to make the request.\u001b[39;00m\n\u001b[1;32m 283\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m 284\u001b[0m session \u001b[38;5;241m=\u001b[39m use_session(session)\n\u001b[1;32m 285\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mcls\u001b[39m(\n\u001b[0;32m--> 286\u001b[0m \u001b[43mvotableparse\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mcls\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_from_result_url\u001b[49m\u001b[43m(\u001b[49m\u001b[43mresult_url\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43msession\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mread\u001b[49m\u001b[43m)\u001b[49m,\n\u001b[1;32m 287\u001b[0m url\u001b[38;5;241m=\u001b[39mresult_url,\n\u001b[1;32m 288\u001b[0m session\u001b[38;5;241m=\u001b[39msession)\n",
"File \u001b[0;32m/opt/lsst/software/stack/conda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/astropy/io/votable/table.py:164\u001b[0m, in \u001b[0;36mparse\u001b[0;34m(source, columns, invalid, verify, chunk_size, table_number, table_id, filename, unit_format, datatype_mapping, _debug_python_based_parser)\u001b[0m\n\u001b[1;32m 159\u001b[0m config[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mfilename\u001b[39m\u001b[38;5;124m\"\u001b[39m] \u001b[38;5;241m=\u001b[39m source\n\u001b[1;32m 161\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m iterparser\u001b[38;5;241m.\u001b[39mget_xml_iterator(\n\u001b[1;32m 162\u001b[0m source, _debug_python_based_parser\u001b[38;5;241m=\u001b[39m_debug_python_based_parser\n\u001b[1;32m 163\u001b[0m ) \u001b[38;5;28;01mas\u001b[39;00m iterator:\n\u001b[0;32m--> 164\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mtree\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mVOTableFile\u001b[49m\u001b[43m(\u001b[49m\u001b[43mconfig\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mconfig\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mpos\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m1\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m1\u001b[39;49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse\u001b[49m\u001b[43m(\u001b[49m\u001b[43miterator\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mconfig\u001b[49m\u001b[43m)\u001b[49m\n",
"File \u001b[0;32m/opt/lsst/software/stack/conda/envs/lsst-scipipe-9.0.0/lib/python3.11/site-packages/astropy/io/votable/tree.py:4148\u001b[0m, in \u001b[0;36mVOTableFile.parse\u001b[0;34m(self, iterator, config)\u001b[0m\n\u001b[1;32m 4145\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mparse\u001b[39m(\u001b[38;5;28mself\u001b[39m, iterator, config):\n\u001b[1;32m 4146\u001b[0m config[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m_current_table_number\u001b[39m\u001b[38;5;124m\"\u001b[39m] \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m0\u001b[39m\n\u001b[0;32m-> 4148\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43;01mfor\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mstart\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtag\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdata\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mpos\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;129;43;01min\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43miterator\u001b[49m\u001b[43m:\u001b[49m\n\u001b[1;32m 4149\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43;01mif\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mstart\u001b[49m\u001b[43m:\u001b[49m\n\u001b[1;32m 4150\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43;01mif\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mtag\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m==\u001b[39;49m\u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mxml\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\n",
"\u001b[0;31mValueError\u001b[0m: 1:0: not well-formed (invalid token)"
]
}
],
"source": [
"dl_results = DatalinkResults.from_result_url(dataLinkUrl,\n",
" session=auth_session)"
]
},
{
"cell_type": "markdown",
"id": "842f11de-df8a-47f3-9a34-12e6a508b248",
"metadata": {},
"source": [
"# Show what is happening\n",
"When this URL is requested, the underlying service responds with a 404 with a helpful error message.\n",
"\n",
"The DataLinkResults class should be surfacing this helpful error message rather than what it is doing."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "bdac3a47-f5b8-4098-92e1-3dc1271a7b25",
"metadata": {
"execution": {
"iopub.execute_input": "2024-11-26T18:18:24.972722Z",
"iopub.status.busy": "2024-11-26T18:18:24.972451Z",
"iopub.status.idle": "2024-11-26T18:18:25.015855Z",
"shell.execute_reply": "2024-11-26T18:18:25.015503Z",
"shell.execute_reply.started": "2024-11-26T18:18:24.972707Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"404\n",
"None\n",
"b'{\"detail\":[{\"loc\":[\"query\",\"id\"],\"msg\":\"Repository for butler://dp02/20d28216-534a-4102-b8a7-1c7f32a9b78c does not exist\",\"type\":\"not_found\"}]}'\n"
]
}
],
"source": [
"from pyvo.dal.query import DALResults\n",
"\n",
"print(DALResults._from_result_url(dataLinkUrl, session=auth_session).status)\n",
"print(DALResults._from_result_url(dataLinkUrl, session=auth_session).msg)\n",
"print(DALResults._from_result_url(dataLinkUrl, session=auth_session).read())\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "LSST",
"language": "python",
"name": "lsst"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment