Mine exploration, photographs and mining history for mine explorers, industrial archaeologists, researchers and historians Mine explorer and mining history videos on YouTube Connect with other mine explorers on Facebook
Tip: do not include 'mine' or 'quarry', search by name e.g. 'cwmorthin', use 'Sounds like search' if unsure of spelling

Advanced Search
'Sounds like search'
Quick a b c d e f g h i j k l m n o p q r s t u v w x y z
Tip: narrow down your search by typing more than one word and selecting 'Search for all words' or 'Exact search'

Search for any word
Search for all words
Exact search
Tip: narrow down your search by typing more than one word and selecting 'Search for all words' or 'Exact search'

Search for any word
Search for all words
Exact search

Mine Exploration Forum

Jump to page << < 1 2 3 > >>
Author Data structure - discussion
SimonRL

Avatar of SimonRL

Joined: 27/11/2005
Location: North Wales

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 12/01/2011 21:25:38
Reply |  Quote
Now then Smile

Following on from the various threads about data, and how it should be arranged, I thought I'd start a new thread to see if we can cut through it all.

I suspect (I'd never have known all this 5 years ago when I started this Wink) that what works for one area of the country might not for another. But hey ho!

The current data structure goes (take mine as a generic term which can include mine, quarry etc.)

mine - link table - album - link table - photo

And then all sorts of others such as:

mine - link table - document

photo - link table - vote

etc.

This is a bit more complicated than simply storing a mine ID value against a photo, but it does mean that a photo could exist in two albums and an album in two mines.

None of that is exactly rocket science, it's fairly fundamental database stuff.

Most of the shortcomings in the current data structure are to inherent limitations in this vis a vis:

1. A number of people have flagged up the issue of changing name over the years of operation of a mine

2. What to do where we've effectively got a parent / child relationship between a level (or whatever) and a mine

So perhaps this structure would be more extensible:

entry *

entry - link table - album *

entry - link table - entry **

entry - link table - entry - album **

So you could have:

Cwmorthin - Photos of Cwmorthin - Photo (demonstrated by *)

Colliery - Pit - Photos of Colliery - Photo (demonstrated by **)

And then at a different branch:

entry - date range - name

It all sounds hellishly complicated. But long term it's about storing all the data in a meaningful format.

Thoughts?

That would be a ground-up rebuild of the entire mine search and and display system, but to be honest that's overdue.

--

Data wise that would be a lot tidier than leaving albumless entries as containers with links in the descriptions to the individual pits, trials and levels etc.

--

Searching (either mine or photo searching) would then need to be in mine names, 'dependant' mine names and alternate names thereof; as well as in date range different names.

Strewth!

--

The only thing I would say is don't expect this overnight Laugh It's probably about 6 months part time work based on current time constraints Sad

I would however rather do it right than shoehorn some nasty kludge in and live to regret that when a) it doesn't work properly and b) I have to do it properly next time...

--

Enough people have over the years said things like "I'd post photos if there was somewhere to put them", this is confusing or that is confusing or there aren't sufficient places to pin things. So perhaps this would address all those issues?

--

Or is this over-complicating things and it's OK as it is?

--

Now, who can I send the bill to Laugh



--

Keep Calm And Carry On
IP: 95.148.101.132 Edited: 12/01/2011 21:39:04 by SimonRL
rikj

Avatar of rikj

Joined: 27/12/2008

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 12/01/2011 21:45:53
Reply |  Quote
Not wishing to create any more work for you Simon, but any chance of showing the possible structure as a drawing? (For those of us who prefer objects to abstracts!)

Or if any other kind soul could oblige Big Grin

My only contribution would be that if something is going to take 6 months to implement, spend at least that long planning it first! I think it's called the 6 Ps rule.

--

sanitas per evolo
IP: 86.128.246.163
JohnnearCfon

Avatar of JohnnearCfon

Joined: 22/12/2005
Location: Sir Caernarfon

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 12/01/2011 23:07:43
Reply |  Quote
I must admit that went right over my head!

Can I request one thing (if it is possible)?

When you are looking at a photo in an album along the top you have:-

Home>nnnn user album>title

Would it be possible to add an extra level in that so you get:-

Home>nnnn Mine>nnnn user album>title

In that way, when navigating, you dont have to go back from the photo to the album if you want to look at info about the mine. That would be very useful if you have arrived at a user album, gone straight to a photo you like then want to look at mine info.

I hope that makes sense. Also hope that isn't part of what you said above!
IP: 78.150.244.16
NewStuff

Avatar of NewStuff

Joined: 26/07/2010
Location: NE Wales

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 12/01/2011 23:19:27
Reply |  Quote
JohnnearCfon wrote:

I must admit that went right over my head!

Can I request one thing (if it is possible)?

When you are looking at a photo in an album along the top you have:-

Home>nnnn user album>title

Would it be possible to add an extra level in that so you get:-

Home>nnnn Mine>nnnn user album>title

In that way, when navigating, you dont have to go back from the photo to the album if you want to look at info about the mine. That would be very useful if you have arrived at a user album, gone straight to a photo you like then want to look at mine info.

I hope that makes sense. Also hope that isn't part of what you said above!


It makes sense to me, and although you don't have to go home to get anywhere, you do need to go back to the User album.
Another level displayed on the "Path" perhaps Simon?

--

In your mines, Taking your pictures...
IP: 86.173.143.84
SimonRL

Avatar of SimonRL

Joined: 27/11/2005
Location: North Wales

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 12/01/2011 23:39:11
Reply |  Quote
NewStuff wrote:

JohnnearCfon wrote:

I must admit that went right over my head!

Can I request one thing (if it is possible)?

When you are looking at a photo in an album along the top you have:-

Home>nnnn user album>title



It makes sense to me, and although you don't have to go home to get anywhere, you do need to go back to the User album.
Another level displayed on the "Path" perhaps Simon?


Yes, absolutely Smile It's just a level of recursion I didn't implement first time around.

But not in this version of the site.

In the next version Wink so back to the data strucure discussion Wink

--

Keep Calm And Carry On
IP: 95.148.11.141
Lister

Avatar of Lister

Joined: 07/10/2007
Location: Helsby, Cheshire

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 12/01/2011 23:49:17
Reply |  Quote


Although I do agree with the last point Shocked

...Lister;~)

--

'Adventure is just bad planning' Roald Amundsen
IP: 92.29.205.210
NewStuff

Avatar of NewStuff

Joined: 26/07/2010
Location: NE Wales

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 13/01/2011 10:32:06
Reply |  Quote
Having re-read your OP with a reasonable amount of coffee, it seems like a realistic option. The "Dependant" entries in particular would clear up a lot of confusion, but I can forsee this needing a fair bit of input from Members into sorting them out in the first place, once the change is implemented.

--

In your mines, Taking your pictures...
IP: 86.173.143.84
SimonRL

Avatar of SimonRL

Joined: 27/11/2005
Location: North Wales

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 13/01/2011 11:41:46
Reply |  Quote
I think that's the key bit to it, it's going to take input from a lot of people to make a really great database.

If the interest isn't there, it won't work.

I guess there's a balance to be struck between making it all about data and getting things correct and making it about getting out and about and exploring!

On reflection the database modifications might be more straight forward than I originally anticipated, mines/sites just need linking to each other in the same way as the major surface feature system works, but with 3 link types:

Served by - e.g. the mill site serving many mines
Served - e.g. the mines served by a mill
Belonging to - e.g. levels and pits belonging to a mine / colliery but not correctly classified as standalone mines

--

Keep Calm And Carry On
IP: 95.148.11.153
royfellows

Avatar of royfellows

Joined: 13/06/2007
Location: Great Wyrley near Walsall

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 13/01/2011 11:58:07
Reply |  Quote
For the benefit of those who struggle to understand the structure of a relational database such as whatever 'back ends' adit now, here is an example. This is the table structure of the Cumbrian Miners database in the MS Access data file. This was developed by me a few years ago for a mine research society. Its purpose is to show how relationships work within a relational database.
If you look at "Mines" and "Miners" you will see that they are related through a 'bridge' table "Mine_Miners", a miner can work at many mines in his lifetime, and a mine can emply many miners. So its a 'one to many' running both ways



(click image to open full size image in new window)



--

''the stopes soared beyond the range of our caplamps' - David Bick...... How times change
IP: 78.150.194.69
BertyBasset

Avatar of BertyBasset

Joined: 13/12/2007
Location: Caernarfon, North Wales

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 13/01/2011 13:39:40
Reply |  Quote
It might be worth taking the object approach rather than dealing directly with database structure.

1. Define the objects of interest e.g. mine, mine company, entrance, mine name, photo, user, document etc.
2. Define relationships between your objects
3. Derive a database structure from your object relationships.

One way of deriving objects is to describe what the web site does (or you want it to do), then extract a list of nouns from your description, then analyse the noun list and throw away repeats and those that are irrelevant. Those left are potential objects.



Apologies if i'm teaching grannies to suck eggs and all that...

Cheers,

Robin
IP: 194.28.139.160
davel

Avatar of davel

Joined: 24/07/2007
Location: Gwynedd

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 13/01/2011 14:12:07
Reply |  Quote
I think another essential is to divorce the database from the user interface.

The database should support the recording (for each mine or quarry) of mine names, location, miners, owners, photographs, plans, whatever i.e. the sort of relational database structure Roy mentions a couple of postings back.

My starting point for the database design would be to consider a mine to be a specific geographical site which has a defined location and other data such as the one or more names it is known by associated with it.

In addition, we probably need some form of data validation when entering a new mine) to ensure that mines which have the same or very similar location (NGR in the UK) and or the same or similar names really are distinct sites and not duplicate entries. Ideally, the system should also force (or at least encourage) a standard case and capitalisation scheme on mine names.

The user interface should be designed to use that data to provide the search and display facilities that the users require.

Ideally, for a given mine one would be able to see when it was worked and who the owners etc. were - then for any owner see what other mines they were involved in. (Now where have I come across that idea before? Big Grin [web link])

So we probably ought to distinguish in this discussion between what the users want (e.g. a request for specific information to be shown on photo pages) and what database structure is required to support that and other requirements.

Dave
IP: 195.137.87.110
SimonRL

Avatar of SimonRL

Joined: 27/11/2005
Location: North Wales

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 13/01/2011 14:12:37
Reply |  Quote
Thanks. This is exactly the sort of feedback I was hoping for.

Basically I'm fine with the data structure when it comes to actually building and interrogating it.

But the schema - that needs input from the people who complain the current one isn't good enough Wink

OK, we have:

mine to album to photo

But that's too restrictive

We need to be able relate mines to mines

And names to period to mines

And be searchable within all

But this is what I need, ideas thrown about so I can build something that works for the next decade and beyond (the current build of the site has lasted 5 years)

I'd rather not rebuild it for the grumblers to start grumble that version 2 still isn't good enough Wink

--

Keep Calm And Carry On
IP: 95.148.11.153
SimonRL

Avatar of SimonRL

Joined: 27/11/2005
Location: North Wales

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 13/01/2011 14:16:54
Reply |  Quote
davel wrote:

Ideally, for a given mine one would be able to see when it was worked and who the owners etc. were - then for any owner see what other mines they were involved in. (Now where have I come across that idea before? Big Grin [web link])


I think that's great, it's really deep content, but it does rather rely on people populating it. There's a finite number of people who will give their time to entering this data. Take the SW, there's 2 people working really hard on that at the moment using all manner of sources. But it's a bit lacking in some parts of the country. What I need is somebody to step forward and say "OK, you build it, I'll make sure enough people can be encouraged to help with the data" Wink Flowers

davel wrote:

So we probably ought to distinguish in this discussion between what the users want (e.g. a request for specific information to be shown on photo pages) and what database structure is required to support that and other requirements.

Dave


I'm hoping I can construct the former around the latter. It doesn't matter from the user's point of view what is behind the scenes, but the behind the scenes bit needs to be able to deliver what the users want Big Grin

--

Keep Calm And Carry On
IP: 95.148.11.153
royfellows

Avatar of royfellows

Joined: 13/06/2007
Location: Great Wyrley near Walsall

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 13/01/2011 14:37:11
Reply |  Quote
davel wrote:

Ideally, the system should also force (or at least encourage) a standard case and capitalisation scheme on mine names.

Dave


I can write a VBA script that will do this, but sorry, I cannot work in any other language.

Similarly, all the work done relative to the "Wheals", I could have wrote a script that would do this in seconds. An SQL action query would be simpler and better.

--

''the stopes soared beyond the range of our caplamps' - David Bick...... How times change
IP: 78.150.194.69
BertyBasset

Avatar of BertyBasset

Joined: 13/12/2007
Location: Caernarfon, North Wales

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 13/01/2011 15:27:21
Reply |  Quote
Actually, as a matter of interest. Do the aspx pages sit directly on data access objects or do you have a 'business' layer for abstracting relationships ?

Robin
IP: 194.28.139.160
SimonRL

Avatar of SimonRL

Joined: 27/11/2005
Location: North Wales

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 13/01/2011 15:30:12
Reply |  Quote
Direct in this version of the site, it harks from an era where I was effectively writing classic ASP inside ASPX pages having only just started the transition. Albeit all data queries are via SP.

I'm not an out and out programmer, just a humble web developer Wink

--

Keep Calm And Carry On
IP: 95.148.11.153 Edited: 13/01/2011 15:33:01 by SimonRL
mountainpenguin

Joined: 18/12/2006

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 13/01/2011 18:47:42
Reply |  Quote
hmm well
if it were me i would be using a gis database (e.g. postgis) to store location data as it can do some rather nice stuff.
I would then use tags to add to add more data information.
you can then get to the data in pretty much any way the gis lets you ask questions like things near other things, in things etc.
The tags let you explore in other interesting aways.
I would also use the meta data as more tags so you can ask questions like show me all pics taken with a d10 or all pics taken between the 9th of feb and 1st of may etc.
IP: 83.67.133.116
SimonRL

Avatar of SimonRL

Joined: 27/11/2005
Location: North Wales

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 13/01/2011 19:24:11
Reply |  Quote
mountainpenguin wrote:

hmm well
if it were me i would be using a gis database (e.g. postgis) to store location data as it can do some rather nice stuff.


Any ideas you can give me on licensing costs? Server requirements and integration with such things as member database, login, tying in with forum, events etc.?

mountainpenguin wrote:

I would then use tags to add to add more data information.
you can then get to the data in pretty much any way the gis lets you ask questions like things near other things, in things etc.
The tags let you explore in other interesting aways.


Not my area I'm afraid, but if you say it'd do it all then I'll take your word for it Smile

mountainpenguin wrote:

I would also use the meta data as more tags so you can ask questions like show me all pics taken with a d10 or all pics taken between the 9th of feb and 1st of may etc.


Is that relevant though? I'd far rather images were captioned with "Cwmorthin back vein chamber x floor y January 2011" than with "Canon D10 f5.8 16sec AWB"

Devil

--

Keep Calm And Carry On
IP: 95.148.102.126 Edited: 13/01/2011 19:35:02 by SimonRL
mountainpenguin

Joined: 18/12/2006

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 13/01/2011 20:02:54
Reply |  Quote
simonrl wrote:

mountainpenguin wrote:

hmm well
if it were me i would be using a gis database (e.g. postgis) to store location data as it can do some rather nice stuff.


simonrl wrote:


Any ideas you can give me on licensing costs? Server requirements and integration with such things as member database, login, tying in with forum, events etc.?


well gis bits are location aware databases.
Have a look at http://postgis.refractions.net/documentation/ and http://www.freegis.org/


mountainpenguin wrote:

I would then use tags to add to add more data information.
you can then get to the data in pretty much any way the gis lets you ask questions like things near other things, in things etc.
The tags let you explore in other interesting aways.


Not my area I'm afraid, but if you say it'd do it all then I'll take your word for it Smile

mountainpenguin wrote:

I would also use the meta data as more tags so you can ask questions like show me all pics taken with a d10 or all pics taken between the 9th of feb and 1st of may etc.


Is that relevant though? I'd far rather images were captioned with "Cwmorthin back vein chamber x floor y January 2011" than with "Canon D10 f5.8 16sec AWB"

Devil


ahh sorry wasnt very clear.
in your example
the user / software would add tags for:
Cwmorthin back vein chamber x floor y
Cwmorthin back vein
back vein
chamber x floor y
floor y
Exploring new ground with simon

the software would get (from the pic its in the exif data so it comes for free)
taken with a d10 focused at infinity 4s exposure on the 11th of jan at 12:23

the software then gets built on top of the metadata 9this is how flickr works)
so you can get all Cwmorthin pics or all Cwmorthin back vein chamber x floor y pics
or all Cwmorthin pics taken on the 11th of jan .....
As a DB structure its pretty simple you have pic id and associated tagid and the tags table with tagid,tagtype and tagvalue.
you can then add more tag types e.g. location, author, camera, fstop .....
The UI then sits on top of an abstraction layer that you can ask questions so adding new UI bits is (relatively) simple ...

In a nutshell I would take flicker and add a custom UI on top to make browsing and uploading a lot more domain specific i.e. fit with mines.

I hope that makes a kind of sense but its probably easier to explain in person.

IP: 83.67.133.116
SimonRL

Avatar of SimonRL

Joined: 27/11/2005
Location: North Wales

View Profile
View Posts
View Personal Album
View Personal Files
View all Photos
Send Private Message
Data structure - discussion
Posted: 13/01/2011 20:10:02
Reply |  Quote
mountainpenguin wrote:

In a nutshell I would take flicker and add a custom UI on top to make browsing and uploading a lot more domain specific i.e. fit with mines.


In which case, and at the risk of a bit of insider trading, can I have a tip-off when you float on the stock market please Wink

mountainpenguin wrote:

I hope that makes a kind of sense but its probably easier to explain in person.


It makes sense, but I'm not sure it is do-able given my skill set and the very clear way those with the knowledge on the subject matter are telling me they want the data structured. Which is more about mines and how they operated and less about photos and exif data.

I think perhaps you'd best explain it in person!

But, I think rebuilding around the current format, adding a few new levels of data-linking and generally sprucing things up and making them easier to use is the way to go... I'm a one man band and if I could build the next Flickr I'd be richer than I'll probably ever be Smile

--

Keep Calm And Carry On
IP: 95.148.102.126 Edited: 13/01/2011 20:11:52 by SimonRL
Jump to page << < 1 2 3 > >>
Safety LED Miners Caplamps Moore Books: Specialist Books I.A. Recordings: Mining and Industrial History DVDs Starless River - Caving Store Explore a Disused Welsh Slate Mine
Disclaimer: Mine exploring can be quite dangerous, but then again it can be alright, it all depends on the weather. Please read the proper disclaimer.
© 2005 to 2015 AditNow.co.uk
Top of Page