AminiTech/amini-fm

Amini FM is an open-weights transformer model for learning representations of satellite pixel time series. It accepts 48 time positions and 10 numerical channels per pixel. Its six-layer encoder uses temporal attention and cross-channel attention, with a hidden width of 512.

The model is part of Amini's work on self-supervised representation learning for Earth observation. It adapts PatchTST to numerical spectral-band and index sequences, learning through reconstruction of randomly masked temporal patches. The released checkpoint contains an encoder and a patch reconstruction head.

Model details

Property	Value
Developer	AminiTech / Amini.ai
Architecture	PatchTSTForPretraining
Model size	Approximately 18.94M
Input shape	[batch_size, 48, 10]
Encoder layers	6
Hidden width	512
Attention heads	8
Feed-forward width	2048
Patch length / stride	12 / 12
Cross-channel attention	Enabled
Normalization	layernorm
Internal scaling	mean
Masking	Random patches; configured ratio 0.4
Reconstruction objective	Mean squared error
CLS token	Enabled
Weight format / precision	Safetensors / float32
Exported Transformers version	4.51.3

Architecture details refer to this checkpoint configuration. Context length counts positions after preprocessing; it does not specify a calendar duration.

Uses

• Generate pixel time-series embeddings for similarity search and clustering.
• Train downstream land-use/land-cover classifiers with labeled data.
• Explore temporal patterns in agricultural and environmental data.

Classification, regression, and forecasting require an appropriate downstream model and task-specific evaluation. The reconstruction head does not produce land-cover labels or future forecasts. The prediction_length field in the configuration does not establish a trained forecasting capability.

Input and preprocessing

Provide finite float32 values with shape [batch_size, 48, 10], preserving chronological order and consistent channel definitions.

Each input channel should have a consistent feature definition and position. Temporal resampling should produce 48 positions per sequence.

Use the channel order, feature definitions, normalization, and missing-data handling associated with the training pipeline for your application. The model applies internal mean scaling.

Training and related research

The exported configuration records self-supervised random patch masking and MSE reconstruction. The mask ratio is a configuration value; the realized fraction depends on the discrete number of patches.

Related research describes Sentinel-2 median biweekly composites from 2020-2024 and 50,000 chips across Africa. The companion study presents a compact channel-independent model; this larger release adds cross-channel attention. Its downstream performance should be evaluated for the intended task. ICLR 2025 ML4RS paper

Usage

Install PyTorch, Transformers with PatchTST support, and Safetensors. The example below uses synthetic input to demonstrate the API. Real inputs require the preprocessing described above.

Extract embeddings

import torch
from transformers import PatchTSTModel

MODEL_ID = "AminiTech/amini-fm"
REVISION = "5ce4e875ad358f4a3f842f33c7d36b8c7dd433ac"

encoder = PatchTSTModel.from_pretrained(
    MODEL_ID,
    revision=REVISION,
    use_safetensors=True,
    do_mask_input=False,
).eval()

past_values = torch.randn(
    1, encoder.config.context_length, encoder.config.num_input_channels,
    dtype=torch.float32,
)
observed_mask = torch.ones_like(past_values, dtype=torch.bool)

with torch.inference_mode():
    hidden = encoder(
        past_values=past_values,
        past_observed_mask=observed_mask,
    ).last_hidden_state

    # Shape: [batch, channels, patch_tokens_including_cls, hidden_width]
    patch_hidden = hidden[:, :, 1:, :]  # Exclude the prepended CLS token.
    per_channel_embeddings = patch_hidden.mean(dim=2)
    pixel_embeddings = per_channel_embeddings.mean(dim=1)

print(pixel_embeddings.shape)  # torch.Size([1, 512])

Loading the bare encoder omits the reconstruction head and may produce an unused-weight warning. This example averages patch embeddings and then channels; pooling should match the downstream training protocol. Channel-level or CLS-token embeddings can also be used.

Run masked reconstruction

from transformers import PatchTSTForPretraining

model = PatchTSTForPretraining.from_pretrained(
    MODEL_ID, revision=REVISION, use_safetensors=True,
).eval()

with torch.inference_mode():
    outputs = model(
        past_values=past_values,
        past_observed_mask=observed_mask,
    )
    reconstruction_loss = outputs.loss

The pretraining class applies random input masking even in evaluation mode. The encoder example explicitly disables masking for repeatable embeddings. Missing values must be represented consistently with the applicable preprocessing; the all-ones mask above is for fully populated synthetic input.

Evaluation and limitations

The usage examples have been checked with the released weights on CPU using PyTorch 2.6.0 and Transformers 4.48.3; this checks loading and forward-pass behavior, not downstream accuracy.

• Clouds, missing observations, interpolation, sensor differences, and feature scaling can affect embeddings.
• Transfer across regions, seasons, sensors, and acquisition conditions requires evaluation on representative data.
• Random pixel splits can contain spatially related examples; geographic and temporal holdouts are useful when assessing generalization.
• Reconstruction loss alone does not establish downstream classification, crop-yield prediction, calibrated change detection, or causal relationships.

License

This checkpoint is released under the Apache License 2.0, as declared in its repository metadata.

References

• Self-Supervised Representation Learning on Remote Sensing Pixel Time Series with Patch-Based Masking, ICLR 2025 ML4RS Workshop.
• Hugging Face PatchTST documentation.