Jan Karel Pieterse
Op 14 mei 2014 organiseren wij in Amsterdam de eerste "Amsterdam Excel Summit". Een absoluut unieke groep Excel MVP's zal in mei 2014 in Amsterdam zijn om hun geweldige Excel kennis met u te delen. Deze MVP's zijn in Amsterdam voor een bijeenkomst en wij zijn erin geslaagd om deze mensen voor ons evenement te boeken. Er is slechts weinig kans dat een dergelijke mogelijkheid zich nogmaals zal voordoen, dus wees er snel bij als u dit niet wilt missen!
Jan Karel Pieterse
Join us in Amsterdam on May 14 2014, for the first Amsterdam Excel Summit. An absolute unique group of Excel MVP's will gather in Amsterdam to share their expert knowledge with you. The Excel MVP's happen to be in Amsterdam for a meeting and we've succeeded in getting some of them to present at our event. There is not much chance on this happening again anytime soon, so make sure you register!
Kelly Sommers wrote a blogpost called 'Create benchmarks and results that have value' in which she refers to my last ORM benchmark post and basically calls it a very bad benchmark because it runs very few iterations (10) and that it only mentions averages (I do not, the raw results are available and referred at in the post I made). Now, I'd like to rectify some of that because it now looks like what I posted is a large pile of drivel.
The core intent of the fetch performance test is to see which code is faster in materializing objects, which framework offers a fast fetch pipeline and which one doesn't, without any restrictions. So there's no restriction on memory, number of cores used, whether or not it utilizes magic unicorns or whether or not a given feature has to be there, everything is allowed, as it just runs a fetch query like a developer would too. If a framework would spawn multiple threads to materialize objects, use a lot of static tables to be able to fetch the same set faster in a second attempt (so memory would spike), that will go unnoticed. This is OK, for this test. It's not a scientific benchmark setup in a lab with dedicated hardware run for weeks, it's a test run on my developer workstation and my server. I never intended it to be anything else than an indication.
The thing is that what I posted, which uses publically available code, is an indication, not a scientific benchmark for a paper. The sole idea behind it is that if code run 10 times is terribly slow, it will be slow a 100 times too. It's reasonable to assume that as none of the frameworks has self-tuning code (there's no .NET ORM which does), nor does the CLR have self-tuning code (unlike the JVM for example). The point wasn't to measure how much slower framework X is compared to framework Y, but which ones are slower than others: are the ones you think are fast really living up that expectation? That's all, nothing more.
That Entity Framework and NHibernate are slower than Linq to SQL and LLBLGen Pro in set fetches is clear. By how much exactly is not determinable by the test results I posted as it depends on the setup. In my situation, they're more than 5-10 times slower, on another system with a different network setup the differences might be less extreme, but the differences will still be there, likewise the difference between the handwritten materializer and full ORMs.
The core reason for that is that the code used in the fetch pipelines is less optimal in some frameworks than in others, or that they perform more features with every materialization step than others. For example, LLBLGen Pro for each row read from the DB, it goes through its Dependency Injection framework, checks for authorizers, auditors, validators etc. to see whether the fetch of that particular row is allowed, converts values if necessary to different types, all kinds of features which can't be skipped because the developer relies on them, however they all add a tiny bit of overhead to each row.
Fair or unfair? Due to this, my ORM will never be faster than Linq to SQL, because Linq to SQL doesn't do all that and has a highly optimized fetch pipeline. If I pull all the tricks I know (and I already do all that), my pipeline will be highly optimal but I still have the overhead of the features I support. Looking solely at the numbers, even if they come from a cleanroom test with many iterations, it doesn't give you a full picture. This is also the reason there's one distinction made between micro-ORMs and full ORMs, and another on change tracked fetches and readonly fetches and for example there's no memory measurement given (also I couldn't do that reliably from code). Again this isn't a scientific paper, it's an indication with a strict definition of its limited scope.
I posted averages, but contrary to what Kelly states, I also posted the full results. The averages were posted as an easier way to illustrate the point of the whole post, namely an indication. The averages were in my opinion acceptable because the results are practically equal across the board; some iteration is a couple of ms slower than the other, but overall they're more or less equal, for the point of the test: the small differences in milliseconds between the iterations are irrelevant for the point of the test: 6686ms in an iteration is slower than 559ms, averaged or not. If there would have been spikes, averages would have been bad, I completely agree there. One problem I had with the previous code was for example the individual fetch tests, which were run on an VM back then. These fetches flooded the VM so much that the results were completely unreliable with high spikes. Averaging those would be stupid, so I made the code wait in such a way that GC would be forced and VM port flooding would be avoided.
Was the averaging bad? For a scientific benchmark it would be bad indeed, but as this isn't a scientific benchmark, it even runs just 10 iterations and ran on my dev box, not a testlab, and as this isn't about exact numbers, I think it's not particularly bad: the averages given are not different from the set they're pulled from (the standard deviation is extremely low), so they can be used as a raw indication whether that framework is particularly slow or not, compared to the others. Which was precisely the point of the test, nothing more.
'But what about her point?'
The general point Kelly tries to make is justified and I agree with that. The thing is that my indication test isn't something on which it applies. I don't have dedicated hardware for this test (which is essential for a scientific benchmark, so you know nothing else interfered) nor do I have time to run a million iterations of the test, as that will takes months, with the code running 24/7. I don't expect the code to give different results btw if one does so, but as it's on github, feel free to do it yourself. The whole point was to give an indication in a very strict scope, and you can verify it with the code yourself.
On closing, I'd like to address one paragraph from Kelly's post.
We don’t know what the various code is doing. Benchmarking helps to find what kind of trade-offs exist in the code and how they impact workload. 10 iterations is not enough to get anything from.
A million iterations (which btw will take more than 67 days alone for Entity Framework) won't give you anything of that either: you still wouldn't know what the ORM is doing under the surface which causes it to be less fast than another one. Linq to SQL would still be faster, Entity Framework would still be dead last. This is simply because there are no moving parts here: after the first iteration, everything is fixed, no self-optimizing runtime, no optimizer, nothing, the exact same steps will be taken by every framework.
If we are benchmarking something running on the CLR, JVM or Go, when do I care about how fast my ORM is 10 times without the GC? Never. I always care about the GC. The GC is there, and this test is too fast to get it involved. I have no idea if any ORM’s in this test are creating a really bad GC collect situation that will result in major performance drops.
This is true, but it's outside the scope of the test, which is strictly defined, and I did that deliberately to avoid things like the above quote. There's no memory test done with the fetch test. It would have been interesting though, e.g. to illustrate the string caching system LLBLGen Pro uses to avoid large amounts of redundant string instances in memory, but alas, it's outside the scope of the test (it still eats some performance though). The GC might have a harder time on the crap left behind by framework X than it has with the garbage left behind by framework Y. The test was deliberately meant to test how efficient the fetch pipeline of a given framework is, the GC therefore isn't taken into account, it's actually forced to run outside the test so it doesn't skew the results.
There are a lot of things not visible in a test this short. I don’t know if any ORM’s eat my RAM for lunch, or if some are extremely CPU intensive. It’s just too short.
It's too short for your view of what this test illustrates, but that wasn't the intention of the test at all. For example, the memory usage measurement can only be done on a dedicated test setup (so dedicated server, network and client for this benchmark), where a test of a single ORM is run for days. The code I wrote doesn't work for that, as multiple ORMs are run side by side: one can ruin the GC generations for others, which skews the results, if the GC is allowed to run. A dedicated setup is then also required, as each ORM is run separately, so to avoid having different test situations, the setup must be equal to all tests.
I write ORMs for many many years now, I know what tricks others use to get their performance and I also know why some of my code isn't as fast as others. This test was a fun (well, it was fun till people start bashing it as if I have no clue) way to see how do the different frameworks relate to each other. I picked a strict scope to avoid as many crap as possible, and I also didn't pick a situation in which my own code would be the sole winner of it all, simply because that wasn't the point of the test to begin with. I wanted an indication how the fetch performance of the various ORMs relate to each other. I still think the results I posted do give a proper indication about that and that was all I wanted to do with this.
As the fun has bled out of it, this will likely be the last post about this topic. I still hope it was helpful to you all.
This is the second post about fetch performance of various .NET ORM / data-access frameworks. The first post, which has lots of background information can be found here. In this second post I'll post new results, including results from frameworks which were included after the previous post. The code used is available on GitHub. I'd like to thank Jonny Bekkum for adding benchmark code for many of the frameworks which were added after the previous post.
In the previous benchmark run, it clearly showed that Entity Framework 6 and NHibernate were, well… slow. I filed a bug with the Entity Framework team and they created a workitem for it for v6.1. They were able to make Entity Framework perform better by 20-30%, but only in the situation where foreign key fields were present in the model. I discovered that that wasn't the case in the previous setup. The performance tests shown today are with foreign key fields present as other frameworks fetch these too and it's only fair that they're present to get a full picture. If they're not included, performance of v6.1 is slightly faster than 6.0.2 (~3000ms in 6.0.2 to ~2830ms in 6.1).
Telerik Data Access
New in this benchmark is, among others, Telerik DataAccess. Jonny added two versions, one using normal domain mappings and one using the code-first Fluent Mappings. These mappings were generated using the Telerik Wizard. However, this wizard creates dreadful code: the generated mappings class contains a vital flaw: without altering the code, it will create the meta-data model every time which obviously is tremendously slow. See the class for details (line 34-41)
Since the previous setup, I've replaced my old server with a new one and have installed SQL Server 2012 on the OS itself instead of in a VM so the results are little better overall than before because of this.
- Server: Windows Server 2012 64bit, SQL Server 2012 on i5-4670 @ 3.4ghz with 8GB ram, raid 1 HDDs.
- Client: Windows 8.0 32bit, .NET 4.5.1 on a Core2Quad @ 2.4ghz with 4GB ram
- Network: 100BASE-T
- Database: AdventureWorks 2008
- Entity Fetched: SalesOrderHeader
For a discussion about this setup and why it is done this way, please see the first post.
The code has been drastically improved since the first post. It's now a properly designed application with a runner and easy to implement benchmark classes so it's easy to add another framework to the system. Please have a look at the code at the github repository.
The raw results can be found here. They include the full output as well as the aggregated results in a nice list. Additionally to the previous benchmark run, individual fetch tests have been added as well as tests how fast enumerating the fetched collection is. This is done to discover which frameworks offload work to when the program actually consumes the result-set which can hide the real performance of the set fetch if this isn't taken into account. The Entity Framework v6.1 results are from a separate batch run before the results seen below, and are solely meant to illustrate progress made in v6.1.
With change tracking (10 runs, averaged, fastest/slowest ignored)
'With change tracking' is as the name implies, a fetch of elements which can be used in read/write scenarios: the changes made to the data are tracked and can be easily persisted. In general this takes some extra work (not much though, if you're clever )
|Framework, with change-tracking ||Fetch avg. (ms) ||Enumeration avg. (ms) |
|DataTable, using DbDataAdapter ||532.63 ||52.00 |
|Linq to Sql v126.96.36.199 (v4.0.30319.18408) ||638.75 ||2.75 |
|LLBLGen Pro v188.8.131.52 (v4.1.14.0117) ||685.25 ||10.75 |
|Telerik DataAccess/OpenAccess Domain v4.0.3 ||1197.38 ||3.00 |
|Telerik DataAccess/OpenAccess Fluent v4.0.3 ||1218.38 ||3.00 |
|Oak.DynamicDb using typed dynamic class ||1304.13 ||1624.88 |
|NHibernate v184.108.40.20600 (v220.127.116.1101) ||3910.13 ||4.00 |
|Entity Framework v18.104.22.168 (v6.1.30207.0) ||4081.63 ||3.00 |
|Entity Framework v22.214.171.124 (v6.0.21211.0) ||6701.38 ||3.00 |
Clearly we see that a typed dynamic class is very slow during enumeration (Oak.DynamicDb enumeration average). For more info about this, see this post by Oak developer Amir Rajan. It also shows that Microsoft made good progress in Entity Framework v6.1 but is still far off from what people expect from it, especially when you think about the fact that Linq to Sql's performance is almost 10 times faster (let that sink in for a minute) than it's bigger brother Entity Framework v6.0.2.
Without change tracking (10 runs, averaged, fastest/slowest ignored)
Without change tracking is a fetch to read-only elements: changes to the data (if possible) are not tracked, so if you use these fetches in read/write scenarios you have a hard time persisting changes made unless you do a lot of work. This usually leads to faster fetches, as the work to make sure changes are tracked can be avoided. This is in general the territory of the Micro-ORMs which are simply there to do one thing: read data as fast as possible into objects and nothing else.
|Framework, without change-tracking ||Fetch avg. (ms) ||Enumeration avg. (ms) |
|Handcoded materializer using DbDataReader ||500.38 ||2.00 |
|PetaPoco Fast v4.0.3 ||502.00 ||2.00 |
|PetaPoco v4.0.3 ||515.88 ||2.00 |
|Dapper ||519.38 ||2.00 |
|Linq to Sql v126.96.36.199 (v4.0.30319.18408) ||561.75 ||2.88 |
|Entity Framework v188.8.131.52 (v6.0.21211.0) ||564.75 ||2.13 |
|ServiceStack OrmLite v184.108.40.206 (v220.127.116.11) ||589.75 ||2.00 |
|LLBLGen Pro v18.104.22.168 (v4.1.14.0117), typed view ||736.00 ||5.00 |
|Oak.DynamicDb using dynamic Dto class ||1269.50 ||199.50 |
Here we see that none of the frameworks can beat the hand-written materializer. This is not a surprise as all these frameworks have to do some work to make sure the data fits in an instance of a type they generate at runtime in some cases. LLBLGen Pro's Typed Views (which are typed datatables here) are not doing so well, even though the DataTable fetch in the previous table was very fast, and it even gets beaten by the change tracked entity fetches. A reason for this is that the projection code currently uses a pipeline which can handle any row at any given moment, but for set fetches which expects each row to have the same amount of fields, it's overhead which can be done without, however this requires architectural changes which I can't make mid-release.
Individual fetches with change tracking (100 individual fetches, 10 runs)
This benchmark fetches change tracking elements (see above) but this time it fetches 100 elements, one element at a time. Although the times taken by most frameworks are very low, it gives another insight in what a framework is doing.
|Framework, with change-tracking, individual fetches ||individual fetch avg. (ms) |
|Telerik DataAccess/OpenAccess Domain v4.0.3 ||0.67 |
|Oak.DynamicDb using typed dynamic class ||0.67 |
|Telerik DataAccess/OpenAccess Fluent v4.0.3 ||0.69 |
|DataTable, using DbDataAdapter ||0.73 |
|LLBLGen Pro v22.214.171.124 (v4.1.14.0117) ||1.19 |
|NHibernate v126.96.36.19900 (v188.8.131.5201) ||1.33 |
|Entity Framework v184.108.40.206 (v6.0.21211.0) ||2.44 |
|Linq to Sql v220.127.116.11 (v4.0.30319.18408) ||2.64 |
Comparing this table with the one which fetches a set, we see a different picture. Here we see the Telerik code performing much better than when fetching a set, which is also true for NHibernate. Linq to Sql was fast with fetching a set but now ends up last.
Individual fetches without change tracking (100 individual fetches, 10 runs)
This benchmark fetches non-change tracked, read-only elements (see above) with the same setup as with the change tracked equivalent: it fetches 100 elements, one element at a time.
|Framework, without change-tracking, individual fetches ||individual fetch avg. (ms) |
|Dapper ||0.58 |
|Oak.DynamicDb using dynamic Dto class ||0.62 |
|ServiceStack OrmLite v18.104.22.168 (v22.214.171.124) ||0.67 |
|Handcoded materializer using DbDataReader ||0.76 |
|PetaPoco Fast v4.0.3 ||0.99 |
|LLBLGen Pro v126.96.36.199 (v4.1.14.0117), typed view ||1.55 |
|Entity Framework v188.8.131.52 (v6.0.21211.0) ||2.19 |
|Linq to Sql v184.108.40.206 (v4.0.30319.18408) ||2.54 |
|PetaPoco v4.0.3 ||3.76 |
Here we see something interesting: some Micro-ORMs are faster than the hand-written materializer. The overhead in the hand-written materializer is done more efficiently by these frameworks it seems. Another thing of note is that PetaPoco, one of the fastest in the set fetch, is not very fast when it comes to fetching single elements. The same is true for Linq to Sql which was also not a great performer in the change tracked bench.
Now, I won't copy the same answers to the obvious questions from the previous post, so if you want to be Captain Obvious again and state something already answered there, please consult the list of remarks already addressed in the previous post.
If you want your framework added to the benchmark, please send a Pull Request. The framework must be used by developers out there.
Jan Karel Pieterse
RefTreeAnalyser 2.0 has just been updated. Prevented opening of unneeded blank workbook. Ever had to work out the logic of other people's Excel files? Ever had to untie the spaghetti-knots of a large Excel workbook's formulas? Then you know what a nightmare this can be! Now there is the RefTreeAnalyser! With this tool, finding out how a cell in a workbook derives its results and what other cells depend on the cell is a breeze.
Jan Karel Pieterse
RefTreeAnalyser 2.0 has just been updated. Made searching in Objects optional for searching precedents and dependents to improve performance. Ever had to work out the logic of other people's Excel files? Ever had to untie the spaghetti-knots of a large Excel workbook's formulas? Then you know what a nightmare this can be! Now there is the RefTreeAnalyser! With this tool, finding out how a cell in a workbook derives its results and what other cells depend on the cell is a breeze.
There has been some talk around several internet outlets about the (seemingly) eroding trust developers have in Microsoft and its techniques (see David Sobeski's piece here, Tim Anderson's piece here and e.g. the Reddit Programming thread here). Trust is the keyword here and in my opinion it's essential to understand what that means in the context of a software developer to understand the problem at hand, or even to acknowledge that there is / isn't a problem. I try to explain below what I think trust means in this context.
Every day some individual or team comes up with a new tool, framework, version of an existing tool / framework, new OS or even a new way to bake those nice chocolate cookies. We as humanity tend to call this progress, even though in many occasions the (small) step taken by said individual or team is actually a (small) step sideways or even backwards. To understand why these individuals and teams still come out in the open to demonstrate their results which sometimes result in total lack of real progress, we first have to understand how product development works and why people even do it.
For every successful product (be it a commercial tool/framework/system, open source system/framework/tool or a cookie recipe, you the idea) out there, there are many many more failures, left for dead in repositories on dying harddrives on local servers or cloud storage. The core reason this happens is that there's no recipe for a successful product: to be successful you first have to create something, ship something. No shipping, no success, and as there's no way to know whether success will come after shipping, all we can do is ship as much as possible so the chances are higher that something is picked up and success eventually comes. The rest of the products are ignored, kicked to the curb and eventually moved to /dev/null.
The motivation to keep doing this of course is the vague 'success' term used in the previous paragraph. 'Success' can mean a lot of different things: become famous, earn lots of money, learn a lot, get a respected title, become respected among peers, the list is endless. All these different things motivate people to create new things, to ship their work in the hope it will give them the success they strive for. Microsoft isn't different in that, as a product producer they have to keep on shipping new, relevant products in the hope these products become successful and with that will make the company money so it lives on.
Microsoft has released a tremendous amount of products in the past decades: from frameworks to languages to operating systems to APIs to word processors. Some were successful, others keeled over right after their shiny boxes appeared on store shelves. The successful ones are continued, the less successful ones are abandoned and replaced with a new attempt to gain success. Over time even successful ones were abandoned or replaced because of a different phenomenon: the interest in the product faded, as illustrated in the technology adoption lifecycle.
So in short, Microsoft behaved like any other product producing individual / team: they produced as many products as possible in the hope that as many of them would become successful. That combined with the technology adoption lifecycle in hand, it's clear that to stay successful one has to keep producing products. That is, from the point of view of the product producer. But what about the product consumer, the person who bought into these products, learned how they worked, behaved, what their limits were, in short took a dependency on them? Is the story the same as the one from the product producer? Sadly, no.
As a software developer we are both product producers (commercial or OSS frameworks/tools/systems or project based custom software) and consumers: we consume frameworks, tools and systems to produce frameworks, tools and systems. To be successful we, like any other product producer, release as many products as we can in the hope that some will gain success. By 'release products' I mean this in the broadest sense: releasing successful custom projects, tools, commercial frameworks or OSS libraries: you shipped software. Shipping custom software written on consultancy basis still contributes to success: the more you ship, the more one will tend to hire you. No-one wants to hire a developer who sticks around for years without shipping anything.
If a product we ship gains success we want to keep that success, so we want to get that bell curve from the technology adoption lifecycle as wide as possible: the wider it is, the longer we can reap the product's success. However there's a problem. To produce this successful product, we took dependencies on the products of others, so our success is tied to the bell curves of those products. If our product can stay successful if the products we took a dependency on stay the same as they are today, the problem is not that big: our clients/customers/users use our product, not the ones we depend on. But what does that mean, 'stay successful'? Is that a given, once the product gets some success?
In most cases, to keep your success you have to keep working on your product: add new features, polish the features already there, remove the ones which don't work anymore, change the looks of the product so it looks new/changed. That way, new users might become interested and take the plunge and will start using your product. As software developers we can only make these changes if the products we depend on let us do so. If the products we depend on are too limiting, it might be the required changes to e.g. make the step from Early Adopters to Early Majority won't be added and the product will never reach the end of the bell curve and will die prematurely.
With writing custom software on consultancy basis this is actually the same thing: there the product you produce is what you can create with the knowledge and skillset you have, those are the dependencies you took. If you are hired to create software which requires knowledge of a product you don't know anything about, you have the same problem as the software producer who creates a software product and who's faced with a framework which is end-of-life: a dependency you took is out of date.
So in the Microsoft world we have on one side the company Microsoft who cranks out products like there's no tomorrow, and on the other side we have developers using these products to create new products, depending on the products Microsoft produces. As soon as Microsoft abandons a product, all developers who took a dependency on that product, either through knowledge for their consultancy work or for producing software with it, inherit a problem through that abandonment: their own products now run the risk of following the same road as the abandoned Microsoft product.
There are several solutions for this: take a dependency on an alternative product, e.g. from a 3rd party vendor/team/individual, rewrite the functionality the dependency was all about yourself, keep on using the (soon to be) outdated product or abandon your own product as well and create a new one. All have their side effects and all have some things in common: they all come with pain, loss of time, risk of losing the success and therefore loss of money. And all that for no gain at all: if the product wasn't abandoned, the software producer could invest the time in the successful product instead, which would benefit the user.
Taking a dependency is therefore a risk: if the product we depend on has a bell curve which doesn't match our own product, i.e. our product's bell curve is wider / ends later, we run the risk that our product will suffer from the fact the product we depend on reaches end-of-life and will be replaced and abandoned. To mitigate that risk, software developers therefore want to align the bell curves of their products with the ones they take a dependency on: if the products they depend on outlive their own products, they have nothing to worry about, give or take some breaking changes along the way.
But how do you know this bell curve of a product you take a dependency on? Well... you don't. Chances are the product producer doesn't even know for sure how the bell curve of the product is or will run: no-one can predict the future and especially not about products. All that is left is trust and hope: trust that the producer of the product you took a dependency on will keep producing the product in a form you can use and hope that the producer stays trustworthy. There's no certainty, only those two: trust and hope.
If the producer of products you depend on becomes untrustworthy, things fall apart: you can't trust them anymore to keep producing the product in a form you can use and therefore you have to acknowledge there's a problem: your own product has a higher risk of ending prematurely unless extra work is done to mitigate the risk. The core issue at play here is: who is doing that extra work.
As mentioned earlier, Microsoft produced and produces a lot of different products on a high speed. This isn't new and the problems for developers with prematurely ending lifecycles isn't either. In that light, one can't say Microsoft has become less trustworthy: if one trusted Microsoft in the past, what's there against not to do that today? However there is something which is new and which wasn't that much of a problem in the past due to monopoly positions: who's going to pick up the inevitable tab when a product is abandoned? Is it Microsoft so it will invest in ease of transition from the abandoned product to the replacement product, or is it the developer who took the dependency on the abandoned product and now has to invest in migrating her/his own product to a replacement product?
In the past, Microsoft did invest sometimes in transition technology but most of the time it didn't and the software developer who took the dependency was forced to invest the time to move to an alternative/replacement product. The developer did this as there was not really a real alternative. However today there are. Plenty. However Microsoft plays the same game as they have in the past: make the transition from abandoned product to a replacement the problem of the developer who took a dependency on the abandoned product, but with the alternatives growing wider in scope and more mature every day, it's not a given anymore that a developer will pick up the tab: if time has to be invested in migrating the product to an alternative, it might as well be wise to migrate the product away from all dependencies on Microsoft products.
The core motivation for doing that is, in my opinion, that there's severe lack of trust and hope that Microsoft takes the responsibility for the effects a premature end-of-life of a product will have, at least for a group of products. Code written against win32 15 years ago will likely still run today and will also do so tomorrow. It's a different story with .NET based applications even when .NET is a virtual machine based system and therefore could abstract away migrations to the products it depends on (underlying system/OS). At the same time the visibility and availability of mature, trustworthy alternatives elsewhere lowers the chance for Microsoft to get away with a lack of responsibility.
In my opinion, trustworthiness is closely tied to how much responsibility you want to take for the lifecycle of your product and thus for the effects the end of the bell curve will have on the people who take a dependency on your product: the more responsibility you're willing to take, the more people will trust you with their dependency as they know the risk for their own products is lower. Microsoft blew this on several fronts and on several occasions in the .NET world.
But is that really anything new? No, it has happened countless times before. In fact, the trust implied by the willingness of developers to pick up the tab in the past is in my opinion an illusion: there was never really any trust, there was just a large 'it is part of the game' acceptance among developers, largely due to lack of knowledge how things were done elsewhere. Compared to those other times, todays world is a different one than the one 10-20 years ago: Microsoft can't get away with it easily anymore: developers know there are other realms, other worlds out there who suffer less from a single company who calls the shots and more importantly: is not willing to face the consequences for developers when products are abandoned.
That's the real problem Microsoft is facing today with respect to developers: will they trust Microsoft enough that they are willing to take a new dependency on the new stuff Microsoft is producing, even though they know the pain will likely be as severe as it is today? Frankly, I doubt it, or at least, the group blindly doing so will be smaller and will do it with less enthusiasm. Simply because there's less reason to.
For the developers who need to take dependencies on Microsoft products, let's hope they realize they need to step up and take responsibility for the effects abandoning products will have on developers. Only then Microsoft will regain the trust it had.
PS: I am fully aware I produce a product which has many customers who take a dependency on it and at the same time has a few dependencies on Microsoft products like .NET, C# and ADO.NET and thus that this essay also applies to my work. The key take away is responsibility: the actions one takes, which effects will these have on the people who took a dependency on your product? I've spent many many months during the past 10-12 years to mitigate those effects as much as possible for my users and will continue to do so, additionally to the offerings like full sourcecode and rich extensibility points for most elements of my system to lower risks for users. This to answer questions people might get when reading the above text.
Jan Karel Pieterse
RefTreeAnalyser 2.0 has just been updated. Fixed a small bug introduced with searching in Objects. Ever had to work out the logic of other people's Excel files? Ever had to untie the spaghetti-knots of a large Excel workbook's formulas? Then you know what a nightmare this can be! Now there is the RefTreeAnalyser! With this tool, finding out how a cell in a workbook derives its results and what other cells depend on the cell is a breeze.
Jan Karel Pieterse
RefTreeAnalyser 2.0 has just been updated. Now if you're tracing precedents or dependents objects (such as pivot tables, charts, data validation, Conditional formatting, ...) are also traced for cell dependencies. Ever had to work out the logic of other people's Excel files? Ever had to untie the spaghetti-knots of a large Excel workbook's formulas? Then you know what a nightmare this can be! Now there is the RefTreeAnalyser! With this tool, finding out how a cell in a workbook derives its results and what other cells depend on the cell is a breeze.
Jan Karel Pieterse
RefTreeAnalyser 2.0 has just been updated. Now you can analyse all your Excel objects (such as pivot tables, charts, data validation, Conditional formatting, ...) for cell dependencies. Ever had to work out the logic of other people's Excel files? Ever had to untie the spaghetti-knots of a large Excel workbook's formulas? Then you know what a nightmare this can be! Now there is the RefTreeAnalyser! With this tool, finding out how a cell in a workbook derives its results and what other cells depend on the cell is a breeze.
Code-first. It's a way of defining mappings for O/R mappers by hand-writing entity classes and then hand-writing mapping files (either by using shortcuts like conventions or by a fluent api which allows you to setup the mappings rather quickly) to a database which might not exist yet. I find using that kind of system rather odd. The thing is that O/R mapping is about an abstract entity definition which is realized in both a class definition and a table/view definition, in such a way that there is a mapping definable between the two definitions (class and table) so instances of the abstract entity definition (the data!) can flow between instances of the two definitions: from a table row to an entity class instance and back or vice versa. The work needed to perform that flow of entity instances is done by an O/R mapper.
Starting with code in the form of entity classes is equally odd as starting with a table: they both require reverse engineering to the abstract entity definition to create the element 'on the other side': reverse engineer the class to the abstract entity definition to create a table and the mappings is equal to reverse engineering a table to a class and create the mappings. What's the core issue is that if you start with a class or a table, you start with the end result of a projection of an abstract entity definition: the class didn't fall out of the sky, it was created after one determined the domain had such a type: e.g. 'Customer', with a given set of fields: Id, CompanyName, an address etc.
What if that abstract entity definition which was used to create the class or table was in a model which contained all of the domain types for the domain used in the software to build? That would avoid a reverse engineering step to get 'the other side' (class if you start from a table, table if you start from a class), and at the same time it would give a couple of benefits: you can create overviews of the model and more importantly, changes in the domain can be applied directly into the model which then ripple through to classes and tables in the right form, without you needing to worry how that should be done correctly, and without reverse engineering steps which might ignore information which was present in the actual model.
Isn't it then rather odd that when serializing 'objects' to JSON, the overall consensus is that the data is serialized, but when the same object is serialized to a table row, it's actually persisted as a whole? If you are still convinced O/R mapping is about persisting objects, what happens with 'your object', persisted to a table row, if that object is read by a different application, which targets the same database, and which doesn't use an O/R mapper? That application, written in an entirely different language even, can perfectly fine read and consume the entity instance stored in the table row, without even knowing you considered it a persisted .NET object. Because, surprise, the contents of the table row isn't a persisted object, it's a persisted entity instance, an instance of an abstract entity definition, not an instance of a class definition.
But let's say you simply want to work with code only, you don't want to look at models other than when they wear swim gear. Code, being text, has some side effects which might not make it the best medium to define domain models in. Code is readable and changeable easily but to get an overview what's going on, a text editor isn't sufficient anymore, tooling is needed to get proper overview how the model looks like, what the associations are, which inheritance relationships are present. One can't do that by simply looking at the code, the code has to be interpreted or better: reverse engineered to a model, to be able provide the information you're looking for. With a small, 10 entity large model in the form of classes this might work, but if you have to work with a more real-life model with 50 or 100 or even more entities, it's not going to be easy at all.
In case you have a hard time grasping how little one can determine from code alone, try to determine how a database looks like when all you have is 50 tables in DDL SQL statements, complete with their unique constraints and foreign key constraints. It's not a surprise to many that years ago database developers already realized that without proper tooling working with larger relational schemas would be a big nightmare. Strange that code-first using developers don't have that problem at all. At least they don't admit it, otherwise they wouldn't be using code-first at all.
Though let's ignore that for a second. Let's say you as a code-first-I-persist-objects kind of developer have the overview of the model in your code by simply looking at a couple of classes. Are you then problem free? Not really. You see, code-first hides the other side of the picture, or better: a construct in code-first might have devastating results for the other side. E.g. some people have the urge to create a common base class / entity for all their entities in which they define fields like 'ModifiedBy', 'CreatedOn' etc. so all entities will have these fields, and they only have to define them once.
However, there's a problem: inheritance in memory with objects is cheap, it's expensive in a relational database, as with every query, joins will be added for all super/subtypes, if the inheritance is in the form of Target per entity. Is this visible in the code? Is this visible elsewhere? Likely not. Code-first systems often provide a way to use shortcuts: to define repetitive constructs once through conventions. This might hide the results done to the relational schema, as it's unclear how the tables might look like and how they'll result in queries at runtime, simply because the way code is constructed for in-memory use is used as the preferred structure for the relational model as well. Starting from tables of course also hides the other side from the picture, however choices in the DB don't have these kind of effects in code, most of the time.
A better choice is to define the abstract entity model first, use model first with this model to produce both sides, so changes determined in the domain will be reflected as changes in the abstract model and through that as changes to the classes and tables. As it's a model, it can be used as such, so it's easier to make changes, to get overviews from various perspectives and to make sure the changes made to the model are reflected correctly in the classes and tables. It's almost 2014, doing arcane work like in the early days of relational databases and DDL SQL scripts is not necessary anymore. Unless you fancy typing everything out, like a human code generator, you know the people who are replaceable with a tool.
I've added an additional test result, namely for Linq to Sql with change tracking switched off (in the answers, at the bottom of the article). I also have updated the graph so it's now partitioned: the frameworks which do change tracking and the ones which don't do change tracking are now grouped together. DbDataAdapter with DataTable is added to the change tracking set, as a DataTable does change tracking.
I've thought long and hard if I should blog about this, but in the end I decided it would benefit everyone if I would post about this and would be as open as possible. This article is about fetch performance, namely how quickly can a given ORM or Data-access framework fetch a given set of data from a database into a set of objects. The results presented here are obtained using publically available code which is run on a specific setup which is explained below. I'll go deeper into the results as well and will try to counter some obvious remarks which are always made when benchmarks are presented.
I'll stress again: this benchmark is solely a result of fetch performance measurement, using simple queries which should barely take any time in the DB itself. If your favorite framework gets slaughtered here because it's very slow, and you simply don't believe it, you can run the tests yourself by cloning the repository linked above.
One thing to keep in mind as well is that the actual raw numbers given are unique for the test setup below here in my office. That's OK, as the actual raw numbers aren't that important; what's important is how the frameworks relate to one another. If the hardware is very fast, the numbers will be lower, but the slowest framework will still be the slowest and the fastest framework will still be the fastest and with the same relative margins.
As this benchmark is about measuring fetch performance, the code ran to fetch data should not be constrained by the database taking away cycles on the system. Therefore the benchmark obtains the data from a database on the network. Nothing used in the setup is particularly new or high-end, except the code of the frameworks of course. Whether a part is not high-end isn't really important either, as every tested framework has the same elements to work with: the same network, the same database, the same .NET framework, the same hardware.
Server: Windows XP, SQL Server 2005, running in a VM with 1 CPU and 1.5GB Ram on VMWare server on a Core2Duo box.
Client: Windows 8.0 32bit, .NET 4.5.1 on a Core2Quad @ 2.4ghz with 4GB ram
I admit, that's not state-of-the-art material, but I assure you, it doesn't matter: all frameworks have to work with this setup, and slowness of a framework will still show itself, relative to the others. If framework X is 5 times slower than fetching the data using a DataTable, it will still be 5 times slower than that on the fastest hardware, as the DataTable fetch will also be much faster on the fastest hardware.
The bottlenecks in the frameworks will show their ugly heads anyway, if the hardware is the latest and greatest or a VM. For the people wondering why I picked XP, it's my old dev partition which I transformed into a VM to keep the complete environment in tact for support for older LLBLGen Pro versions, in this case v1 and v2, and it happens to have a good AdventureWorks example DB installed. The query spends barely 2ms in the DB, so it's more than capable to serve our tests.
As database I've chosen to use AdventureWorks on SQL Server. It has a reasonable amount of data to work with, and more than a couple of tables, so it is not a simple demo / mickey mouse database but closer to a database used in real-life scenarios.
The test consists of fetching all 31465 rows from a single table, SalesOrderHeader. This table was chosen because it has more than a few columns and a variety of types, so the materializers have to work more than with a 3-column table with only integer values. It also has a couple of relationships with other tables which makes it a good candidate to see whether the ORM used has a problem with relationships in the context of a set fetch when entities were mapped on these tables.
I also chose this table because the amount of data isn't that small but also not massive either, so a good average test set for performance testing: if there are slow parts in the fetch pipeline of a framework, this kind of entity will put the spotlight on them easily.
All tests were run on .NET 4.5.1 using release builds. No ngen was used, just the default assemblies. The database was warmed up as well as the CLR with fetches which results were thrown away. For the averages the fastest and the slowest times were discarded.
The included ORM / Data-access frameworks
I've included the following ORM / Data-access frameworks and have specified my reasons why:
- LLBLGen Pro v4.1. Reason for inclusion is because I wrote it and this benchmark was started to see how fast my fetch logic improvements were stacking up against the competition.
- Entity Framework v6, latest release build from nuget. Reason for inclusion is because Microsoft presents it as the standard for data-access in .NET so it's fair to include it to see whether it is indeed the standard to beat when it comes to fetch performance.
- NHibernate v220.127.116.11, latest release build from nuget. Reason for inclusion is because still a lot of people use it and it was once the most used ORM on .NET.
- Linq to Sql. Latest version included in .NET 4.5.1. Reason for inclusion is because it was Microsoft's first ORM shipped for .NET and still a go-to data-access framework for many people as it's simple and highly optimized.
- Dapper. Version from October 2013. Reason for inclusion is that it is considered (one of) the fastest Micro-ORMs on .NET.
- DbDataAdapter with DataTable. Latest version included in .NET 4.5.1. Reason for inclusion is because it's a core part of how to do data-access in .NET for a lot of developers still, who work with typed datasets and stored procedures, and also because of it's speed of fetching sets of tabular data.
- Hand-coded fetch with DbDataReader and a hand-written POCO. Latest version of the DbDataReader code included in .NET 4.5.1. Reason for inclusion is because it shows how fast hand-written, hand-optimized code is and thus in theory how much performance a full ORM framework spills elsewhere on features, overhead or slow code in general.
The DataTable and hand-written fetches are more or less guidelines to see what's to expect and how close the other frameworks come to these two's results. They give a good insight in what's possible and thus show that e.g. expecting way faster fetches is not reasonable as there's not much more to gain than a tight loop around a DbDataReader. In the results you'll see that Dapper did manage to get very close and in the raw results you'll see it sometimes was faster, which is a remarkable achievement.
All used ORM frameworks have the full AdventureWorks model mapped as entities, all tables have a corresponding entity class mapped and all relationships are modeled as well. No inheritance is modeled, as not all of the frameworks support it (there are some inheritance relationships present in the AdventureWorks schema) and the used entity isn't in an inheritance hierarchy so it didn't make much sense to add it.
The raw results are located here. I've replicated the average times below in tabular and graph form. Each operation consisted of fetching 31465 entities from the database with one query and materialize these in individual objects which were stored into a collection. The average times given are the averages of 10 operations per framework, where the slowest and fastest operation were ignored.
|ORM / Data-access framework ||Average time (ms) |
|Entity Framework v6 ||3470.22 |
|Entity Framework v6 with AsNoTracking() ||689.75 |
|Linq to Sql ||746.88 |
|LLBLGen Pro v4.1 ||786.89 |
|LLBLGen Pro v4.1 with resultset caching ||311.50 |
|Dapper ||600.00 |
|NHibernate 18.104.22.168 ||4635.00 |
|Hand-coded into custom objects ||587.00 |
|DbDataReader into DataTable ||598.38 |
The results are not pretty, at least not for Entity Framework and especially not for NHibernate. The micro-ORM Dapper and the DataTable / hand-written fetch code are very fast, as expected, all ORMs are slower than those three. The resultset caching result, which by far the fastest, fetches the resultset once and then re-materializes the objects from that set, so it still loops through the raw rows obtained from the database to materialize new entities, it just doesn't have to go to the DB again and doesn't suffer from the network / DB latency.
A couple of things stand out. I'll address them below.
- NHibernate is extremely slow with fetching a set of entities. When I add just one entity type to the model (SalesOrderHeader) instead of the full model, average is around 1500ms, so there's definitely something going on with respect to having a model with relationships like AdventureWorks and fetching entities in NHibernate. I did a profile run to see whether I made a terrible mistake somewhere or that the NHibernate team messed up big time. The short answer is: it's not me. See below for more details.
- Entity Framework is also extremely slow. I reported this to the Entity Framework team some time ago. They have acknowledged it and have created a workitem for this. It turns out that this issue is also present in Entity Framework 5. When there's just 1 entity in the model, Entity Framework manages to get close to 1100ms average, so similar to NHibernate there's something going on with respect to relationships in a model and fetching. The AsNoTracking call bypasses all benefits of the ORM and simply converts the rows into objects, like Dapper does, and shows what potential speed Entity Framework has under the hood, if it doesn't have to do anything expected from a good ORM.
- Linq to Sql and LLBLGen Pro are close. I am very pleased with this result, considering the amount of work I've spent on optimizing the query pipeline in the past years. I'm also pleased because the amount of features our framework has doesn't make it much slower than a framework which offers much less, like Linq to Sql. To materialize an entity, LLBLGen Pro has to do the following for each row:
- call into the Dependency Injection sub system to inject objects
- call into a present authorizer to test whether the fetch of this row is allowed
- call into a present auditor to log the fetch of a given row
- call into the uniquing sub system to make sure no duplicates are materialized
- call into the conversion sub system to convert any db values to specific .net types if type conversions are defined.
- call into the string cache to avoid duplicate instances of the same string in memory to avoid memory bloat.
- create a new entity class instance and store the values of the entity into it.
What's also interesting (not shown) is that if there's just one entity in the Linq to Sql model, it's very close to Dapper's speed. So a bigger model also causes a speed penalty there, however not as big as with Entity Framework or NHibernate.
- With a somewhat slow connection to the DB, it can be efficient to cache a resultset locally. With faster network connections, this is of course mitigated to a certain point.
Answers to some expected remarks / questions
Below I'll try to answer as many questions you might have after reading this as well as some remarks which are to be expected. This isn't my first benchmark I'm doing in the long time I'm writing ORM frameworks.
The framework I work full time on is indeed included in the test. Though I didn't skew the test to make it win as I picked a reasonable scenario which will show performance bottlenecks in code used in a lot of LoB apps every day. I also disclosed the full sourcecode archive on github so you can see for yourself what I used for testing and whether I made terrible mistakes.
"Would you have published the results if your own framework was dead last?"
I don't think so, I am honest about that. But perhaps another person would have. This benchmark was started to see whether a prototype I made had a chance to be successful; LLBLGen Pro v3.5 code was much slower than the results offered by v4.1 (around 1280ms on average), and I was pleased with the work done to make fetching data much faster in v4.0. It took months to get there, but the changes were worth it: from 1280 on average to 790 on average was better than I'd have hoped for. To show the world what my hard work had resulted in, I published these results, to inform database using developers that reality is likely different from what they think it is.
"Why this particular test? No-one else tests this"
This test is nothing new. In the earlier days of .NET ORMs we used these kind of fetch benchmarks all the time, especially because earlier on Microsoft pushed typed datasets and stored procedures which leaned a lot on the tabular data fetch performance of the DbDataAdapter. To get close to that performance as an ORM, one had to optimize the pipeline a lot. It's showing that Entity Framework and also NHibernate teams have gotten lazy in this respect that they never tested fetch performance of their frameworks in this way. A solid, good fetch performance is essential for an ORM to be taken seriously by a developer as every bottleneck in an ORM will be a potential bottleneck in the application using it. Performance testing is therefore key for good quality control. Personally I'm very surprised that the two biggest ORMs on .NET perform so poorly.
"I never fetch that much entities, so it doesn't apply to me"
Actually it does. While you might not fetch 31000 entities in one go, but 10000 times 3 entities gives you the same performance degradation: if your application does a lot of fetching, every time a fetch is performed it slows the application down if the fetch itself is slow. If this happens in an application which is relying on good data-access performance, it can bog down the application under load if the data-access performance is sub-par or downright poor, when it didn't have to be.
"Linq to Sql also has a mode where you can switch off change tracking"
Linq to Sql with changetracking disabled was determined after the rest of the article was already written so it was unfair to add that as-is without updating all the numbers again, so instead I'll mention briefly these numbers here: new run, Linq to Sql normal: 738.63ms, Linq to Sql, no change tracking: 649.25ms.
"What should I learn from these results?"
Firstly you can see what the relative performance is of framework X compared to framework Y in this particular situation of fetching a set of entities. Secondly, you can see that there is a hard limit on what performance you can expect from a data-access framework: reading data into entity objects takes time, no matter what you use. This also means that if your application needs to read data faster than e.g. Dapper can give it to you, you likely will not be able to solve that problem in a regular way. Thirdly, if your application feels 'slow' when obtaining data from the database while you're using a framework which is actually pretty fast with fetching data, it might very well be the slowness is elsewhere and not in the data-access / ORM framework used.
"I wrote my own ORM, it's likely faster"
I'm doing this now for a very long time and I've dealt with many developers who wrote ORMs in the past decade or so. Unless you've invested a lot of time in optimizing the fetch pipeline, chances are you made all the rookie mistakes we all made in the beginning. At least that's what I've learned and seen in the past 10-12 years writing ORMs for .NET. But if you're confident, fork the github code and add your solution to it so others can see how quick your code is.
"You didn't include caching for NHibernate"
I indeed didn't as the test wasn't to show how fast caching is in various systems, but how fast entity materialization is. An entity cache could help with NHibernate but it wouldn't make the slow performance of fetching entities any less slow. I included the resultset caching results of LLBLGen Pro to show how fast it is in this context as it does materialize entities again, but it's not the main focus of the benchmark.
"NHibernate is very fast, you messed up somewhere"
Well, I knew they weren't the fastest, but I too didn't expect them to be this slow. So to be certain it wasn't me who made a terrible mistake somewhere, I profiled the NHibernate query execution using dotTrace 4.5.1. One look at the profile results explains it: the code internally is so over-engineered, there's no real overview what's going on as it calls code all over the place and it calls a lot of methods a lot of times: there are 31K entities in this set, yet it manages to call a rather expensive method over half a million times. Here's a screen shot of the trace. It's a 'warm call', so pure fetching, no mapping file import done anymore: (click to see a full image)
Having high performance means you have to engineer for that too. It doesn't come out of the box. If you never profile your code, you don't know what's going on at runtime, especially with code that's so fragmented in methods all over the place like with NHibernate. If someone from the NHibernate team wants the dotTrace profile snapshot file (15MB), let me know.
"You made a mistake with framework X"
Please show me what I did wrong and send me a pull request so I can correct it.
"Did you take into account start-up times of framework X?"
Yes, the slowest and fastest time for any framework are ignored, so startup time isn't taken into account in the averages.
"Why didn't you include framework Y? It's way faster!"
I tried to include the most used frameworks, and there are not a lot left anymore. I could have added some more but I leave that to the owners of the frameworks to do so via the github repository.
"Isn't it true that the less features you have, the faster you can be?"
Fetching entities fast is about a couple of things: creating an object to store data in should be very fast, and overhead per row and per field value should be very low. It's tricky to get all three down to the bare minimum in generic code but one way to do it is to do very little or nothing at all per row or per field. As soon as you add more features, each row is affected and this adds up. This is then extrapolated into the results of fetching many rows. So having less features means less overhead per row means faster fetch speeds. This doesn't mean a fast framework is feature-less, though it might. I know that I will never get LLBLGen Pro as fast as Dapper, simply because Dapper does way less things than LLBLGen Pro does with each entity materialization cycle, but I try
"Lies, damn lies and benchmarks"
True, benchmarks often turn out to be equal to lies or marketing if they're given without a context, without all the surrounding information about what's exactly benchmarked, and why. As I mentioned earlier, the results in this test are not usable without the context in which they're created: it's a single test, highlighting a single feature, performed on a single setup. The numbers only have meaning with respect to that setup, however, the relative performance differences are usable outside the setup: Entity Framework is ~5 times slower than LLBLGen Pro when it comes to fetching sets of entities, however how slow exactly in milliseconds depends on the particular setup your application runs on.
I hope this all helps developers out there open their eyes how fast some of the frameworks out there really are with respect to entity fetch operations. If you have more questions, please post them in the replies so I can answer them.
We've released LLBLGen Pro v4.1! Below a quick run down of what's new in this release. LLBLGen Pro v4.1 is a free upgrade for v4.x licensees.
- Entity Framework v6 support. Entity Framework v6 is now a supported framework, on .NET 4, .NET 4.5 and .NET 4.5.1
- .NET 4.5.1 support .NET 4.5.1 is now a supported platform.
- VS.NET 2013 integration. The designer now integrates also in VS.NET 2013.
- Oracle 12c support. The Oracle drivers now support the new Oracle 12c identity/default sequence feature.
- ODP.NET 12c managed provider support. The ODP.NET based Oracle driver can now work with the ODAC 12+ managed provider.
- Various small changes and additions. See for the full list of smaller changes and additions the online documentation.
LLBLGen Pro Runtime Framework
- Full async API. The runtime framework now offers a full Async API, usable in Linq, QuerySpec and the Low-Level API for retrieval of data and object persistence. Usable in async/await code on .NET 4.5 and higher. More information...
- Transient Error Recovery. The runtime framework now offers the ability to recover from transient errors occuring during database activity. More information...
- Oracle 12c support. The runtime now supports Oracle 12c: Identity/default sequences are now supported on Oracle 12c as well as the new paging keywords for SELECT on Oracle 12c.
- Managed Oracle ODP.NET provider support. The runtime now supports the managed ODP.NET provider for Oracle (ODAC v12+).
- Various small changes. See for the full list of smaller changes the online documentation
Entity Framework support
- Entity Framework v6 support. In v4.1 of LLBLGen Pro, Entity Framework support has been updated to support Entity Framework v6 on .NET 4, .NET 4.5 and .NET 4.5.1.
Jan Karel Pieterse
RefTreeAnalyser 2.0 has just been updated. Improved placement of pictures of far-away cells. Ever had to work out the logic of other people's Excel files? Ever had to untie the spaghetti-knots of a large Excel workbook's formulas? Then you know what a nightmare this can be! Now there is the RefTreeAnalyser! With this tool, finding out how a cell in a workbook derives its results and what other cells depend on the cell is a breeze.
Jan Karel Pieterse
RefTreeAnalyser 2.0 has just been updated. Visualisation colors can now be changed. Ever had to work out the logic of other people's Excel files? Ever had to untie the spaghetti-knots of a large Excel workbook's formulas? Then you know what a nightmare this can be! Now there is the RefTreeAnalyser! With this tool, finding out how a cell in a workbook derives its results and what other cells depend on the cell is a breeze.