Azure table storage allows you to store different data structures in the same table, but the standard routines for getting data in and out require you to know that structure at compile time – as you have to define the entity (class) to be persisted.
There are some very clever implementations that use the dynamic object features of C# to create objects with run-time defined properties. But what if you just want to store “some additional data” in key/value pairs but you don’t really need them to be properties on an object? There’s nothing out of the box for that, but you can override that by altering the Read/WriteEntity methods on TableEntity.
Here’s a naive implementation (it assumes string data, doesn’t limit number of data items, etc.)
public class TestEntity : TableEntity {
public TestEntity(string a, string b) {
PartitionKey = a;
RowKey = b;
DataItems = new Dictionary<string, string>();
DataItems.Add("foo", "bar");
DataItems.Add("Jacob", "Xander");
}
public Dictionary<string, string> DataItems { get; set; }
public override IDictionary<string, EntityProperty> WriteEntity(OperationContext operationContext) {
var results = base.WriteEntity(operationContext);
foreach (var item in DataItems) {
results.Add("D_" + item.Key, new EntityProperty(item.Value));
}
return results;
}
public override void ReadEntity(IDictionary<string, EntityProperty> properties, OperationContext operationContext) {
DataItems = new Dictionary<string, string>();
foreach (var item in properties) {
if (item.Key == "Timestamp") {
Timestamp = item.Value.DateTimeOffsetValue.Value;
} else if (item.Key == "RowKey") {
RowKey = item.Value.StringValue;
} else if (item.Key == "PartitionKey") {
PartitionKey = item.Value.StringValue;
} else if (item.Key.StartsWith("D_")) {
string realKey = item.Key.Substring(2);
ItemData[realKey] = item.Value.StringValue;
}
}
}
}
First key thing to remember – Azure table storage can only store a total of 255 key/value pairs – or 252 custom ones once you’ve taken PartitionKey, etc. into account.
I’d welcome any comments, improvements, or just drop me a note if you’ve used this in your own project – thanks!
Great walk through, can you just correct the title typo hastable -> hashtable
Good spot, corrected, thank you!
Good post. Out of interest, why not use Blob Storage if you’re just after key/value pairs? That way you don’t have the 1Mb limit.
IIRC, we wanted to store both the incoming data (which was in various flat file formats) and the results of the initial processing of those files into records – for e.g., a CSV came in and we stored that in Blob Storage. We then processed that CSV into records and stored each row/document individually in Table Storage. There was more than just “read the line” logic to the parsing, and our data was well under 1Mb per row.
The reasoning was that as the incoming file format changed (and it would), we didn’t need to have a parser that could read each version in the live code – we could just have the live parser running at all times and then use Table queries to look at how historic data was read down at the row-by-row level. It could also cope with changing from a CSV to an API call and all historic data could still be read from the same data store. It could also be queried by any application that can talk to Table Storage rather than being limited to just our app.
It’s possible, though, that Blob Storage has moved on a bit since I was working with it and now supports some level of querying, I’d be interested to hear if that’s the case!
please have a look at the new methods TableEntity.Flatten and TableEntity.ConvertBack in Azure Storage SDK v8.0.0.
https://msdn.microsoft.com/en-us/library/mt775435.aspx
https://msdn.microsoft.com/en-us/library/azure/mt775432.aspx
Looks very relevant – I’ve long left this project but if I have something similar to do in the future, thanks for the pointer!