What should an RF data lake actually do?

Storage is not the same as a data lake

An RF data lake should do more than store files.

That sounds obvious, but it is where a lot of RF data conversations get blurry. Teams talk about storage, buckets, shared drives, archives, and data lakes as if they are the same thing.

They are not.

Storage gives the data a place to live. An RF data lake should make the data usable after it gets there.

For RF teams, that distinction matters because the files are usually not self-explanatory. A large IQ capture may be useful, but only if the team can answer basic questions later: what was collected, where it came from, what settings were used, whether it has already been processed, who can access it, and whether it is worth opening in the first place.

Ingest is where a capture becomes manageable

A useful RF data lake should start with ingest.

That means getting large captures into the system without turning every upload into a one-off project. The system should accept the files teams already have, preserve the original capture, and make room for metadata that may come from the collection system, a sidecar file, an operator note, or analysis that happens later.

Ingest is where the capture becomes something the team can manage instead of a file that merely moved from one place to another.

Metadata has to be signal-aware

Metadata is the next piece.

A filename might tell you the date or project. That helps, but it is not enough. RF teams usually need signal-relevant fields: center frequency, sample rate, bandwidth, timestamp, source, sensor configuration, test event, environment, and notes about how the capture was collected.

Some metadata will be clean. Some will be incomplete. Some will arrive later. The data lake should handle that reality instead of assuming every capture arrives perfectly labeled.

Search should match how engineers remember data

Search has to use that metadata.

Finding RF data by folder path or object key works only until the dataset gets large enough to forget. Engineers should be able to search by the fields they actually remember: a frequency range, a time window, a sensor, a test event, a capture type, or whether related analysis exists.

A good search workflow reduces the number of people who have to ask, 'Where did we put that file?'

Preview comes before deeper analysis

Preview matters too.

Large IQ captures are expensive to move around. If an engineer has to download the full file just to decide whether it is relevant, the system is already wasting time. A quick spectrogram preview can answer the first question before deeper analysis starts: is this probably the signal, time range, or capture I need?

Preview does not replace analysis. It keeps teams from doing unnecessary work before analysis.

Storage efficiency cannot break the workflow

Storage efficiency still matters.

RF datasets can grow quickly, especially when captures are duplicated across shared drives, project folders, scratch space, and analysis directories. Compression can help, but only if it fits the workflow. A smaller file is useful when teams can still search it, preview it, control access, move it when needed, and connect it to analysis outputs.

The goal is not compression for its own sake. The goal is to reduce storage burden without making the data harder to use.

Access, audit, and analysis history belong in the data layer

Access control and audit history are part of the data layer.

RF data often has different sensitivity levels depending on source, collection context, program, customer, or environment. The system should help teams control who can see, download, modify, or share a capture. It should also keep a useful record of what happened: who accessed the data, when it moved, and which derived products came from which original capture.

That matters in normal engineering work. It matters more in secure labs, test ranges, and defense or government environments where auditability and controlled data movement are part of the job.

Analysis outputs need to stay connected. A raw capture is often only the start. Teams create spectrograms, annotations, filtered files, detections, notebooks, reports, and other derived products. If those outputs drift away from the original capture, reproducibility gets weaker. Months later, someone may have the result but not the trail that explains how it was produced.

An RF data lake should help preserve that chain.

Deployment model is part of the product

Deployment model matters as well.

Some teams can use public cloud services. Others cannot. Some need local deployment. Some need air-gapped operation. Some need to run inside a restricted network with strict access policies. For those teams, deployment is not an afterthought. It is part of the product.

So what should an RF data lake actually do?

It should make large captures easier to ingest, describe, search, preview, store efficiently, control, audit, and reuse. It should keep the capture and its context together. It should work with the storage layer underneath without pretending storage alone solves the workflow.

That is the direction we are taking with SigDrive.

SigDrive is being built as an RF data lake for teams that need large captures to remain findable, understandable, and reusable after the original collection is over.

If your team is trying to manage RF data across shared drives, object storage, local machines, scripts, and analysis tools, we would like to compare notes.