Datasetpublic

AINovice2005/carbon-likelihood-stats

Dataset Summary Dataset Summary AINovice2005/carbon-likelihood-stats is a model-enriched dataset derived from the Carbon genomic corpus, containing sequence-level likelihood statistics for 251,427 sequence regions. Each row links a source sequence record to its corresponding region in the processed/tokenized Carbon corpus using record_id, start, and end. The dataset summarizes both the model's overall likelihood of the observed sequence and the variation in… See the full description on the dataset page: https://huggingface.co/datasets/AINovice2005/carbon-likelihood-stats.

sourceHugging Faceapache-2.0updated 8d agoView on Hugging Face

0likes622downloads

Dataset Card

Dataset Summary

AINovice2005/carbon-likelihood-stats is a model-enriched dataset derived from the Carbon genomic corpus, containing sequence-level likelihood statistics for 251,427 sequence regions.

Each row links a source sequence record to its corresponding region in the processed/tokenized Carbon corpus using record_id, start, and end. The dataset summarizes both the model's overall likelihood of the observed sequence and the variation in token-level model confidence within that sequence.

The dataset contains aggregate likelihood measures including mean_log_prob, sum_log_prob, and perplexity, together with local likelihood statistics such as min_token_logprob, argmin_position, and per_token_logprob_std. supervised_position_count records the number of token positions that contributed to the likelihood calculation. The start and end fields represent corpus coordinates.

Intended Uses

This dataset is intended for model-based analysis of genomic sequence likelihoods, including:

—Sequence likelihood distribution analysis
—Sequence ranking and filtering
—Model-based quality assessment
—Detection of sequences with unusual likelihood profiles
—Identification of locally low-probability regions
—Corpus sampling and curation
—Comparison of model likelihoods across Carbon corpus subsets
—Supporting downstream genomic dataset construction and enrichment

The likelihood statistics are intended as model-derived analytical features and should not be interpreted as independent biological or clinical annotations.

Information of Features

Feature	Type	Description
`record_id`	string	NCBI Identifier linking the likelihood statistics to the corresponding source sequence record.
`string_lengths`	int64	Length of the corresponding source sequence/string.
`start`	int64	Starting token position of the sequence in the processed/tokenized Carbon corpus.
`end`	int64	Ending token position of the sequence in the processed/tokenized Carbon corpus.
`mean_log_prob`	float32	Mean log probability assigned by the model to the evaluated tokens. Less-negative values indicate that, on average, the model assigns higher probability to the observed tokens.
`sum_log_prob`	float32	Sum of token-level log probabilities across the evaluated sequence. It represents the aggregate log-likelihood and is naturally affected by sequence length.
`perplexity`	float32	Exponential form of average negative log-likelihood. Lower perplexity means the model finds the sequence more predictable.
`supervised_position_count`	int64	Number of token positions that actually contributed to the likelihood calculation. Useful for understanding the effective amount of scored sequence.
`min_token_logprob`	float32	Lowest token-level log probability encountered in the sequence. It identifies the most surprising/least probable token according to the model.
`argmin_position`	int64	Position of the token having `min_token_logprob`. This identifies where the strongest local likelihood anomaly occurs.
`per_token_logprob_std`	float32	Standard deviation of token-level log probabilities. It measures how much model confidence varies along the sequence.

Intended Uses

This dataset is intended for model-based analysis of genomic sequence likelihoods, including:

—Sequence likelihood distribution analysis
—Sequence ranking and filtering
—Model-based quality assessment
—Detection of sequences with unusual likelihood profiles
—Identification of locally low-probability regions
—Corpus sampling and curation
—Comparison of model likelihoods across Carbon corpus subsets
—Supporting downstream genomic dataset construction and enrichment