{ "cells": [ { "cell_type": "markdown", "id": "7fb27b941602401d91542211134fc71a", "metadata": { "id": "intro-quick-start", "language": "markdown" }, "source": [ "# Quick Start\n", "\n", "Notebook-style examples, using real `nmdc_api_utilities` modules.\n", "\n", "---" ] }, { "cell_type": "markdown", "id": "acae54e37e7d407bbb7b55eff062a284", "metadata": { "id": "ex1-overview", "language": "markdown" }, "source": [ "## Example 1: Study -> Biosamples -> Data Objects\n", "\n", "**Goal:** Start from an NMDC study name, retrieve matching study metadata records. Then collect all linked biosamples and data objects' metadata records that are related to that study.\n", "\n", "### Step 1: Find matching study records" ] }, { "cell_type": "code", "execution_count": null, "id": "9a63283cbaf04dbcab1f6479b197f3a8", "metadata": { "id": "ex1-step1", "language": "python" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Studies found: 1\n", "Study ID: nmdc:sty-11-prtb4s28\n" ] } ], "source": [ "from nmdc_api_utilities import StudySearch\n", "\n", "study_client = StudySearch()\n", "\n", "study_name = (\n", " \"Molecular mechanisms underlying changes in the temperature sensitive \"\n", " \"respiration response of forest soils to long-term experimental warming\"\n", ")\n", "\n", "studies = study_client.get_record_by_attribute(\n", " attribute_name=\"name\",\n", " attribute_value=study_name,\n", " exact_match=True,\n", ")\n", "\n", "print(f\"Studies found: {len(studies)}\")\n", "if studies:\n", " print(f\"Study ID: {studies[0]['id']}\")" ] }, { "cell_type": "markdown", "id": "8dd0d8092fe74a7c96281538738b07e2", "metadata": { "id": "ex1-step2-text", "language": "markdown" }, "source": [ "### Step 2: Get linked biosamples\n", "\n", "If at least one study is found, use the first study ID to request linked biosample records. The `get_linked_instances` method called like this will return a list of fully hydrated biosample metadata records. Note that the `get_linked_instances` method can be used to retrieve linked records of any type, not just biosamples, that are associated with any list of IDs (not just study IDs). It is available on both the `StudySearch` and `NMDCSearch` clients, as well as all the additional search clients that are available in `nmdc_api_utilities`." ] }, { "cell_type": "code", "execution_count": 55, "id": "72eea5119410473aa328ad9291626812", "metadata": { "id": "ex1-step2-code", "language": "python" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Biosamples found: 89\n", "Biosample IDs collected: 89\n", "Example biosample record: \n", "{'id': 'nmdc:bsm-11-eyyxz863', 'type': 'nmdc:Biosample', '_downstream_of': ['nmdc:sty-11-prtb4s28'], 'associated_studies': ['nmdc:sty-11-prtb4s28'], 'collection_date': {'has_raw_value': '2010-06-13', 'type': 'nmdc:TimestampValue'}, 'depth': {'has_numeric_value': 0, 'has_raw_value': '0', 'has_unit': 'm', 'type': 'nmdc:QuantityValue'}, 'description': 'Defining the functional diversity of the Arabidopsis rhizosphere microbiome', 'ecosystem': 'Host-associated', 'ecosystem_category': 'Plants', 'ecosystem_subtype': 'Rhizosphere', 'ecosystem_type': 'Roots', 'elev': 0.0, 'env_broad_scale': {'has_raw_value': 'ENVO_00000446', 'term': {'id': 'ENVO:00000446', 'name': 'terrestrial biome', 'type': 'nmdc:OntologyClass'}, 'type': 'nmdc:ControlledIdentifiedTermValue'}, 'env_local_scale': {'has_raw_value': 'PO_0025025', 'term': {'id': 'PO:0025025', 'name': 'root system', 'type': 'nmdc:OntologyClass'}, 'type': 'nmdc:ControlledIdentifiedTermValue'}, 'env_medium': {'has_raw_value': 'ENVO_00005801', 'term': {'id': 'ENVO:00005801', 'name': 'rhizosphere', 'type': 'nmdc:OntologyClass'}, 'type': 'nmdc:ControlledIdentifiedTermValue'}, 'geo_loc_name': {'has_raw_value': 'USA: North Carolina', 'type': 'nmdc:TextValue'}, 'gold_biosample_identifiers': ['gold:Gb0144817'], 'habitat': 'Arabidopsis rhizosphere', 'host_name': 'Arabidopsis', 'host_taxid': {'has_raw_value': 'NCBITaxon:3701', 'term': {'id': 'NCBITaxon:3701', 'type': 'nmdc:OntologyClass'}, 'type': 'nmdc:ControlledIdentifiedTermValue'}, 'img_identifiers': ['img.taxon:3300012491'], 'insdc_biosample_identifiers': ['biosample:SAMN07632064'], 'lat_lon': {'has_raw_value': '35.9076 -79.0506', 'latitude': 35.9076, 'longitude': -79.0506, 'type': 'nmdc:GeolocationValue'}, 'location': 'USA', 'name': 'Arabidopsis rhizosphere microbial communities from North Carolina - M.Oy.8.old.040610', 'ncbi_taxonomy_name': 'rhizosphere metagenome', 'samp_name': 'M.Oy.8.old.040610', 'samp_taxon_id': {'has_raw_value': 'rhizosphere metagenome [NCBITaxon:939928]', 'term': {'id': 'NCBITaxon:939928', 'name': 'rhizosphere metagenome', 'type': 'nmdc:OntologyClass'}, 'type': 'nmdc:ControlledIdentifiedTermValue'}, 'sample_collection_site': 'Arabidopsis rhizosphere', 'specific_ecosystem': 'Unclassified', 'provenance_metadata': {'type': 'nmdc:ProvenanceMetadata', 'add_date': '2016-12-03T00:00:00Z', 'mod_date': '2024-10-15T00:00:00Z'}}\n" ] } ], "source": [ "biosamples = []\n", "biosample_ids = []\n", "\n", "if studies:\n", " study_id = studies[0][\"id\"]\n", " biosamples = study_client.get_linked_instances(\n", " ids=[study_id],\n", " types=[\"nmdc:Biosample\"],\n", " hydrate=True,\n", " )\n", " biosample_ids = [record[\"id\"] for record in biosamples if \"id\" in record]\n", "\n", "print(f\"Biosamples found: {len(biosamples)}\")\n", "print(f\"Biosample IDs collected: {len(biosample_ids)}\")\n", "print(f\"Example biosample record: \\n{biosamples[0] if biosamples else 'No biosamples found'}\")" ] }, { "cell_type": "markdown", "id": "8edb47106e1a46a883d545849b8ab81b", "metadata": { "id": "ex1-step3-text", "language": "markdown" }, "source": [ "### Step 3: Get linked data objects\n", "\n", "Use biosample IDs as seeds for another linked-instances query targeting `nmdc:DataObject`. For this example, we'll just pull from a subset (5) of the biosample IDs to avoid pulling too many records, but in practice you could pull all linked data objects if desired." ] }, { "cell_type": "code", "execution_count": 56, "id": "10185d26023b46108eb7d9f57d49d2b3", "metadata": { "id": "ex1-step3-code", "language": "python" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Data objects found: 0\n" ] } ], "source": [ "data_objects = []\n", "\n", "if biosample_ids:\n", " data_objects = study_client.get_linked_instances(\n", " ids=biosample_ids[0:5], # Just using the first 5 biosample IDs for this example\n", " types=[\"nmdc:DataObject\"],\n", " hydrate=True,\n", " )\n", "\n", "print(f\"Data objects found: {len(data_objects)}\")\n", "if data_objects:\n", " print(f\"\\nExample data object \\nID: {data_objects[0].get('id')}\")\n", " print(f\"Name: {data_objects[0].get('name')}\")\n", " print(f\"Description: {data_objects[0].get('description')}\")\n", " print(f\"Link for downloading: {data_objects[0].get('url')}\")" ] }, { "cell_type": "markdown", "id": "8763a12b2bbd4a93a75aff182afb95dc", "metadata": { "id": "ex2-overview", "language": "markdown" }, "source": [ "---\n", "\n", "## Example 2: Data Object Type -> Data Objects -> Biosample Metadata\n", "\n", "**Goal:** Start from a `data_object_type` value, retrieve matching data objects, then resolve associated biosample metadata. See schema documentation for more details on the `data_object_type` property and its allowed values: https://microbiomedata.github.io/nmdc-schema/FileTypeEnum/.\n", "\n", "### Step 1: Retrieve data objects by type" ] }, { "cell_type": "code", "execution_count": 57, "id": "7623eae2785240b9bd12b16a66d81610", "metadata": { "id": "ex2-step1", "language": "python" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Data objects found: 25\n", "Data object IDs collected: 25\n" ] } ], "source": [ "import json\n", "\n", "from nmdc_api_utilities import DataObjectSearch\n", "\n", "data_object_client = DataObjectSearch()\n", "\n", "data_object_type = \"Metagenome Raw Reads\"\n", "filter_str = json.dumps({\"data_object_type\": data_object_type})\n", "\n", "data_objects = data_object_client.get_record_by_filter(\n", " filter=filter_str,\n", " all_pages=False, # Set to True to retrieve all matching records across all pages of results\n", ")\n", "data_object_ids = [record[\"id\"] for record in data_objects if \"id\" in record]\n", "\n", "print(f\"Data objects found: {len(data_objects)}\")\n", "print(f\"Data object IDs collected: {len(data_object_ids)}\")" ] }, { "cell_type": "markdown", "id": "7cdc8c89c7104fffa095e18ddfef8986", "metadata": { "id": "ex2-step2-text", "language": "markdown" }, "source": [ "### Step 2: Resolve linked biosample IDs\n", "\n", "Build a mapping from each data object ID to biosample IDs using linked instances." ] }, { "cell_type": "code", "execution_count": 58, "id": "b118ea5561624da68c537baed56e602f", "metadata": { "id": "ex2-step2-code", "language": "python" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Objects with biosample links: 25\n", "Unique biosample IDs: 25\n" ] } ], "source": [ "associations = {}\n", "biosample_ids = []\n", "\n", "if data_object_ids:\n", " associations = data_object_client.get_linked_instances_and_associate_ids(\n", " ids=data_object_ids,\n", " types=[\"nmdc:Biosample\"],\n", " hydrate=False,\n", " )\n", " for data_object_id in data_object_ids:\n", " associations.setdefault(data_object_id, [])\n", "\n", " biosample_ids = sorted(\n", " {\n", " biosample_id\n", " for linked_ids in associations.values()\n", " for biosample_id in linked_ids\n", " }\n", " )\n", "\n", "print(f\"Objects with biosample links: {len(associations)}\")\n", "print(f\"Unique biosample IDs: {len(biosample_ids)}\")" ] }, { "cell_type": "markdown", "id": "938c804e27f84196a10c8828c723f798", "metadata": { "id": "ex2-step3-text", "language": "markdown" }, "source": [ "### Step 3: Fetch biosample metadata and attach by data object\n", "\n", "Retrieve biosample records and build a per-data-object metadata mapping. Similar to the `get_linked_instances` method, the `get_records_by_id` method is also available across multiple clients in `nmdc_api_utilities` and can be used to retrieve fully hydrated metadata records for any list of IDs, even if those IDs are not linked to each other or do not belong to a common collection." ] }, { "cell_type": "code", "execution_count": 59, "id": "504fb2a444614c0babb325280ed9130a", "metadata": { "id": "ex2-step3-code", "language": "python" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Biosamples fetched: 25\n", "Objects with mapped biosamples: 25\n" ] } ], "source": [ "biosample_records = []\n", "if biosample_ids:\n", " biosample_records = data_object_client.get_records_by_id(ids=biosample_ids)\n", "\n", "biosamples_by_id = {\n", " record[\"id\"]: record\n", " for record in biosample_records\n", " if \"id\" in record\n", "}\n", "\n", "biosamples_by_data_object = {}\n", "for data_object_id in data_object_ids:\n", " linked_ids = associations.get(data_object_id, [])\n", " biosamples_by_data_object[data_object_id] = [\n", " biosamples_by_id[biosample_id]\n", " for biosample_id in linked_ids\n", " if biosample_id in biosamples_by_id\n", " ]\n", "\n", "print(f\"Biosamples fetched: {len(biosample_records)}\")\n", "print(f\"Objects with mapped biosamples: {len(biosamples_by_data_object)}\")" ] }, { "cell_type": "markdown", "id": "59bbdb311c014d738909a11f9e486628", "metadata": { "id": "field-links", "language": "markdown" }, "source": [ "---\n", "## Search client selection and schema field names\n", "\n", "The above examples use the `DataObjectSearch` and `StudySearch` clients because the initial filtering was targeted to a specific collection metadata records (data_object_set and study_set, respectively). To help orient yourself to which client to use for a given query, you can refer to the [typecode-to-class map in the NMDC Schema documentation](https://microbiomedata.github.io/nmdc-schema/typecode-to-class-map/), which shows which schema classes are associated with each typecode and therefore which clients will be able to filter by those schema fields.\n", "\n", "For additional query recipes and MongoDB-style filters, see the Filters page in this documentation set." ] } ], "metadata": { "kernelspec": { "display_name": ".venv (3.12.0)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.0" } }, "nbformat": 4, "nbformat_minor": 5 }