MySQL support

bkuschak commented

2017-04-24 02:04:40 +00:00

(Migrated from github.com)

I’m sending you a pull request for some changes to the Celestial database. Its really rough. Not actually meant for you to pull, but more just to get the conversation started, so you can see what I changed.

Actually I haven’t had time to play with Celestial until now. My ISP changed my IP address and I wasn’t able to access it. I found another way to grab the DB using skyvia.com.

For various reasons I wanted to get this DB running locally, using a MySQL backend rather than Azure. Well, it seems to be working, and it’s pretty fast. Way faster than my company database hosted on some beefy MSSQL server. (But this stuff is for hobby not work.)

So, when you have time, have a look at the diff:

I pulled your Celestial DB and exported it as CSV.
Python script to convert this CSV, and your views, to MySQL syntax and import to a local DB.
New DbLib file to use it.

As a side note, I noticed a gzipped version of the full DB is only 1.6MB. Might actually be feasible to check that in, I don’t know.

A few questions for you:

I understand why the main Component.sql file was converted to git-lfs. But why were the Views/*.sql also converted? These are small files.
I am unable to clone the repo when I have git lfs tracking the .sql files. It fails to download them, (presumably due to quota), and then the rest of the clone fails. Had to turn off lfs tracking to get it to succeed. So bottom line, I can’t see your latest Views/*.sql files.
So I branched off of your July 2016 commit that still had the Views in sql in the repo. I don’t imagine they have changed much (if it all) since then.

I’m sending you a pull request for some changes to the Celestial database. Its really rough. Not actually meant for you to pull, but more just to get the conversation started, so you can see what I changed. Actually I haven’t had time to play with Celestial until now. My ISP changed my IP address and I wasn’t able to access it. I found another way to grab the DB using skyvia.com. For various reasons I wanted to get this DB running locally, using a MySQL backend rather than Azure. Well, it seems to be working, and it’s pretty fast. Way faster than my company database hosted on some beefy MSSQL server. (But this stuff is for hobby not work.) So, when you have time, have a look at the diff: - I pulled your Celestial DB and exported it as CSV. - Python script to convert this CSV, and your views, to MySQL syntax and import to a local DB. - New DbLib file to use it. As a side note, I noticed a gzipped version of the full DB is only 1.6MB. Might actually be feasible to check that in, I don’t know. A few questions for you: - I understand why the main Component.sql file was converted to git-lfs. But why were the Views/*.sql also converted? These are small files. - I am unable to clone the repo when I have git lfs tracking the .sql files. It fails to download them, (presumably due to quota), and then the rest of the clone fails. Had to turn off lfs tracking to get it to succeed. So bottom line, I can’t see your latest Views/*.sql files. - So I branched off of your July 2016 commit that still had the Views in sql in the repo. I don’t imagine they have changed much (if it all) since then.

vidavidorra commented

2017-04-25 21:12:46 +00:00

(Migrated from github.com)

@bkuschak Please see #7 where a discussion is going on about additional database hosting and some other stuff.

issus commented

2017-05-06 07:48:22 +00:00

(Migrated from github.com)

Finally had a chance to look at this.. I'm going to pass on this pull for now, as the desktop util will be the way to go for keeping things in sync. There's web services which create a generic view definition from the Azure views which will allow relatively easy creation of views into different database platforms such as MySQL, SQLITE, or even Access.

I really want to get a local database sync tool built into the desktop app, and much of the serverside groundwork has been laid for that to happen. Altium is so very inefficient at using the database for BOM generation, so a local database can do a BOM in minutes rather than potentially hours.

The latest views are now up to date, I've dropped components.sql from the repo - its not like it was ever able to be cloned in any case... I had originally put the database up as a zip or 7zip (i forget) which was small, but I had complaints that it wasn't diffed by github, so people wanted the full SQL file... and well, then the library got popular and it blew my LFS quota out as soon as it reset.

Finally had a chance to look at this.. I'm going to pass on this pull for now, as the desktop util will be the way to go for keeping things in sync. There's web services which create a generic view definition from the Azure views which will allow relatively easy creation of views into different database platforms such as MySQL, SQLITE, or even Access. I really want to get a local database sync tool built into the desktop app, and much of the serverside groundwork has been laid for that to happen. Altium is so very inefficient at using the database for BOM generation, so a local database can do a BOM in minutes rather than potentially hours. The latest views are now up to date, I've dropped components.sql from the repo - its not like it was ever able to be cloned in any case... I had originally put the database up as a zip or 7zip (i forget) which was small, but I had complaints that it wasn't diffed by github, so people wanted the full SQL file... and well, then the library got popular and it blew my LFS quota out as soon as it reset.

yldzmuhammed commented

2018-01-07 01:55:52 +00:00

(Migrated from github.com)

Finally i have installed your libraries. I have some parts to add to the library. Is there a general rule?And i dont see any pull req here other than this. Are you not allow this?If one to add some parts, what should i do?And how can i do this?
Generaly i know how can i edit library. But never work on a dblib before.

Finally i have installed your libraries. I have some parts to add to the library. Is there a general rule?And i dont see any pull req here other than this. Are you not allow this?If one to add some parts, what should i do?And how can i do this? Generaly i know how can i edit library. But never work on a dblib before.

issus commented

2019-12-11 05:26:00 +00:00

(Migrated from github.com)

V2 library has moved on a long way since the last commits. We compared MySQL to MSSQL and for the backing data it was just too slow, some queries that took under 100milliseconds for MSSQL took over 20 seconds for MySQL on identical hardware. MySQL is not a database I have an interest in supporting going forwards.

bkuschak commented

2019-12-11 16:36:44 +00:00

(Migrated from github.com)

Regarding the performance issues, I’ve been running this database on MySQL with very acceptable performance, but I did have to make some server configuration tweaks and add indexes. Afterwards the speed increased over 60X for some operations.

Details here:
https://github.com/bkuschak/altium-library/blob/bk_mysql/DB/altium_library/README.md

On Dec 10, 2019, at 9:26 PM, Mark notifications@github.com wrote:

V2 library has moved on a long way since the last commits. We compared MySQL to MSSQL and for the backing data it was just too slow, some queries that took under 100milliseconds for MSSQL took over 20 seconds for MySQL on identical hardware. MySQL is not a database I have an interest in supporting going forwards.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

Regarding the performance issues, I’ve been running this database on MySQL with very acceptable performance, but I did have to make some server configuration tweaks and add indexes. Afterwards the speed increased over 60X for some operations. Details here: https://github.com/bkuschak/altium-library/blob/bk_mysql/DB/altium_library/README.md > On Dec 10, 2019, at 9:26 PM, Mark <notifications@github.com> wrote: > > V2 library has moved on a long way since the last commits. We compared MySQL to MSSQL and for the backing data it was just too slow, some queries that took under 100milliseconds for MSSQL took over 20 seconds for MySQL on identical hardware. MySQL is not a database I have an interest in supporting going forwards. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub, or unsubscribe.

bretonjustin commented

2019-12-11 16:43:54 +00:00

(Migrated from github.com)

Will everybody have to install MySQL or only the host?
Somebody could host the serve at home and we connect to it, or we buy a MySQL cloud hosting.

Will everybody have to install MySQL or only the host? Somebody could host the serve at home and we connect to it, or we buy a MySQL cloud hosting.

bkuschak commented

2019-12-11 16:48:13 +00:00

(Migrated from github.com)

I run it locally for performance reasons and so I can work without a network connection. You can certainly run it on a cloud host or somewhere else and connect remotely.
On Wednesday, December 11, 2019, 08:43:56 AM PST, Justin Breton notifications@github.com wrote:

Will everybody have to install MySQL or only the host?
Somebody could host the serve at home and we connect to it, or we buy a MySQL cloud hosting.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

I run it locally for performance reasons and so I can work without a network connection. You can certainly run it on a cloud host or somewhere else and connect remotely. On Wednesday, December 11, 2019, 08:43:56 AM PST, Justin Breton <notifications@github.com> wrote: Will everybody have to install MySQL or only the host? Somebody could host the serve at home and we connect to it, or we buy a MySQL cloud hosting. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Design review closed

This design review cannot be reopened because the branch was deleted.

MySQL support #8

Design review closed