Quick Start

Notebook-style examples, using real nmdc_api_utilities modules.

Example 1: Study -> Biosamples -> Data Objects

Goal: Start from an NMDC study name, retrieve matching study metadata records. Then collect all linked biosamples and data objects’ metadata records that are related to that study.

Step 1: Find matching study records

[1]:

from nmdc_api_utilities import StudySearch

study_client = StudySearch()

study_name = (
    "Molecular mechanisms underlying changes in the temperature sensitive "
    "respiration response of forest soils to long-term experimental warming"
)

studies = study_client.get_record_by_attribute(
    attribute_name="name",
    attribute_value=study_name,
    exact_match=True,
)

print(f"Studies found: {len(studies)}")
if studies:
    print(f"Study ID: {studies[0]['id']}")

Studies found: 1
Study ID: nmdc:sty-11-8ws97026

Step 2: Get linked biosamples

If at least one study is found, use the first study ID to request linked biosample records. The get_linked_instances method called like this will return a list of fully hydrated biosample metadata records. Note that the get_linked_instances method can be used to retrieve linked records of any type, not just biosamples, that are associated with any list of IDs (not just study IDs). It is available on both the StudySearch and NMDCSearch clients, as well as all the additional search clients that are available in nmdc_api_utilities.

[2]:

biosamples = []
biosample_ids = []

if studies:
    study_id = studies[0]["id"]
    biosamples = study_client.get_linked_instances(
        ids=[study_id],
        types=["nmdc:Biosample"],
        hydrate=True,
    )
    biosample_ids = [record["id"] for record in biosamples if "id" in record]

print(f"Biosamples found: {len(biosamples)}")
print(f"Biosample IDs collected: {len(biosample_ids)}")
print(f"Example biosample record: \n{biosamples[0] if biosamples else 'No biosamples found'}")

Biosamples found: 42
Biosample IDs collected: 42
Example biosample record:
{'id': 'nmdc:bsm-11-042nd237', 'type': 'nmdc:Biosample', '_downstream_of': ['nmdc:sty-11-8ws97026'], 'name': 'BW-C-7-M', 'associated_studies': ['nmdc:sty-11-8ws97026'], 'env_broad_scale': {'type': 'nmdc:ControlledIdentifiedTermValue', 'has_raw_value': 'forest biome [ENVO:01000174]', 'term': {'id': 'ENVO:01000174', 'type': 'nmdc:OntologyClass', 'name': 'forest biome'}}, 'env_local_scale': {'type': 'nmdc:ControlledIdentifiedTermValue', 'has_raw_value': 'mineral horizon [ENVO:03600011]', 'term': {'id': 'ENVO:03600011', 'type': 'nmdc:OntologyClass', 'name': 'mineral horizon'}}, 'env_medium': {'type': 'nmdc:ControlledIdentifiedTermValue', 'has_raw_value': 'forest soil [ENVO:00002261]', 'term': {'id': 'ENVO:00002261', 'type': 'nmdc:OntologyClass', 'name': 'forest soil'}}, 'samp_name': 'BW-C-7-M', 'collection_date': {'type': 'nmdc:TimestampValue', 'has_raw_value': '2017-05-24'}, 'depth': {'type': 'nmdc:QuantityValue', 'has_raw_value': '.02 -.10', 'has_maximum_numeric_value': 0.1, 'has_minimum_numeric_value': 0.02, 'has_unit': 'm'}, 'ecosystem': 'Environmental', 'ecosystem_category': 'Terrestrial', 'ecosystem_subtype': 'Temperate forest', 'ecosystem_type': 'Soil', 'elev': 302.0, 'env_package': {'type': 'nmdc:TextValue', 'has_raw_value': 'soil'}, 'geo_loc_name': {'type': 'nmdc:TextValue', 'has_raw_value': 'USA: Massachusetts, Petersham'}, 'growth_facil': {'type': 'nmdc:ControlledTermValue', 'has_raw_value': 'field_incubation'}, 'lat_lon': {'type': 'nmdc:GeolocationValue', 'has_raw_value': '42.481016 -72.178343', 'latitude': 42.481016, 'longitude': -72.178343}, 'samp_store_temp': {'type': 'nmdc:QuantityValue', 'has_raw_value': '-80 Celsius', 'has_numeric_value': -80.0, 'has_unit': 'Cel'}, 'specific_ecosystem': 'Mineral soil', 'store_cond': {'type': 'nmdc:TextValue', 'has_raw_value': 'frozen'}, 'analysis_type': ['metatranscriptomics', 'metagenomics', 'natural organic matter', 'metaproteomics', 'metabolomics', 'lipidomics'], 'gold_biosample_identifiers': ['gold:Gb0157193', 'gold:Gb0158505']}

Step 3: Get linked data objects

Use biosample IDs as seeds for another linked-instances query targeting nmdc:DataObject. For this example, we’ll just pull from a subset (5) of the biosample IDs to avoid pulling too many records, but in practice you could pull all linked data objects if desired.

[3]:

data_objects = []

if biosample_ids:
    data_objects = study_client.get_linked_instances(
        ids=biosample_ids[0:5],  # Just using the first 5 biosample IDs for this example
        types=["nmdc:DataObject"],
        hydrate=True,
    )

print(f"Data objects found: {len(data_objects)}")
if data_objects:
    print(f"\nExample data object \nID: {data_objects[0].get('id')}")
    print(f"Name: {data_objects[0].get('name')}")
    print(f"Description: {data_objects[0].get('description')}")
    print(f"Link for downloading: {data_objects[0].get('url')}")

Data objects found: 531

Example data object
ID: nmdc:dobj-14-1e9qeq49
Name: Hawkes_neg.ref
Description: Natural organic matter negative electrospray mode reference calibration file for Calibration object nmdc:calib-14-hhn3qb47
Link for downloading: https://nmdcdemo.emsl.pnnl.gov/nom/reference_calibration_files/Hawkes_neg.ref

Example 2: Data Object Type -> Data Objects -> Biosample Metadata

Goal: Start from a data_object_type value, retrieve matching data objects, then resolve associated biosample metadata. See schema documentation for more details on the data_object_type property and its allowed values: https://microbiomedata.github.io/nmdc-schema/FileTypeEnum/.

Step 1: Retrieve data objects by type

[4]:

import json

from nmdc_api_utilities import DataObjectSearch

data_object_client = DataObjectSearch()

data_object_type = "Metagenome Raw Reads"
filter_str = json.dumps({"data_object_type": data_object_type})

data_objects = data_object_client.get_record_by_filter(
    filter=filter_str,
    all_pages=False,  # Set to True to retrieve all matching records across all pages of results
)
data_object_ids = [record["id"] for record in data_objects if "id" in record]

print(f"Data objects found: {len(data_objects)}")
print(f"Data object IDs collected: {len(data_object_ids)}")

Data objects found: 25
Data object IDs collected: 25

Step 2: Resolve linked biosample IDs

Build a mapping from each data object ID to biosample IDs using linked instances.

[5]:

associations = {}
biosample_ids = []

if data_object_ids:
    associations = data_object_client.get_linked_instances_and_associate_ids(
        ids=data_object_ids,
        types=["nmdc:Biosample"],
        hydrate=False,
    )
    for data_object_id in data_object_ids:
        associations.setdefault(data_object_id, [])

    biosample_ids = sorted(
        {
            biosample_id
            for linked_ids in associations.values()
            for biosample_id in linked_ids
        }
    )

print(f"Objects with biosample links: {len(associations)}")
print(f"Unique biosample IDs: {len(biosample_ids)}")

Objects with biosample links: 25
Unique biosample IDs: 25

Step 3: Fetch biosample metadata and attach by data object

Retrieve biosample records and build a per-data-object metadata mapping. Similar to the get_linked_instances method, the get_records_by_id method is also available across multiple clients in nmdc_api_utilities and can be used to retrieve fully hydrated metadata records for any list of IDs, even if those IDs are not linked to each other or do not belong to a common collection.

[6]:

biosample_records = []
if biosample_ids:
    biosample_records = data_object_client.get_records_by_id(ids=biosample_ids)

biosamples_by_id = {
    record["id"]: record
    for record in biosample_records
    if "id" in record
}

biosamples_by_data_object = {}
for data_object_id in data_object_ids:
    linked_ids = associations.get(data_object_id, [])
    biosamples_by_data_object[data_object_id] = [
        biosamples_by_id[biosample_id]
        for biosample_id in linked_ids
        if biosample_id in biosamples_by_id
    ]

print(f"Biosamples fetched: {len(biosample_records)}")
print(f"Objects with mapped biosamples: {len(biosamples_by_data_object)}")

Biosamples fetched: 25
Objects with mapped biosamples: 25

Search client selection and schema field names

The above examples use the DataObjectSearch and StudySearch clients because the initial filtering was targeted to a specific collection metadata records (data_object_set and study_set, respectively). To help orient yourself to which client to use for a given query, you can refer to the typecode-to-class map in the NMDC Schema documentation, which shows which schema classes are associated with each typecode and therefore which clients will be able to filter by those schema fields.

For additional query recipes and MongoDB-style filters, see the Filters page in this documentation set.