pandas – SQL Ryan

Converting two-column DataFrame to a dictionary in Python

If you have a Pandas dataframe like the following (if you already have a DataFrame based on a query result, you can use that):

import pandas as pd
data = [['tom', 10], ['nick', 15], ['juli', 14]]
df = pd.DataFrame(data, columns = ['Name', 'Age'])

I wanted a dictionary where I could provide the name (“tom”, “nick”, or “juli”) and receive their age as the result. To convert your DataFrame to a dictionary like this, you’d use the following command:

my_dict = pd.Series(df.Age.values,index=df.Name).to_dict()
print(my_dict)

Python dict with three key:value pairs

Now that you have a dictionary, you can do lookups or whatever else you want. Make sure you take note above that the first parameter uses “values” where the second one doesn’t. If you use values in the second parameter, or you just try to use the “.todict()” function on the DataFrame directly, without converting it to a series first as above, you can end up with Dictionaries with an entry for each column, or dictionaries that contain dictionaries, or a number of other things that don’t work.

Short and to the point – this took me about an hour to find the answer to and hopefully you find it before losing that much time!

Clearing up Windows Cluster terminology

I wanted to clear up some terminology around Windows Clusters that seems to cause a bit of confusion. I’ve stumbled across a few questions on StackOverflow and Experts-Exchange that seem to have some basic confusion around “clustering servers” and “how to install an application to a cluster”, and I’m hoping to set a few things straight.

There’s really no such thing as a clustered server. Servers can have clustering enabled and configured, but the servers themselves aren’t really clustered – they’re just set up to enable clustered applications. When servers are part of a cluster, they still do all their thinking on their own, including running their own applications, services, and tasks, without the other servers in the cluster even being aware.
You don’t cluster servers, you cluster applications and resources. Once servers have had clustering installed and are configured, you can cluster an application or a resource. This clustering is really just telling the cluster manager that you want it to control which server clients talk to when they want to access the resource. The cluster manager ensures that the application (or service or resource) is running on only one node at any given time, and to the extent it’s able, it ensures that it’s always running (watching for a failure and bringing the resource online on another node and then directing clients to that node instead).
Applications don’t have to be “cluster-aware” to be clustered. I work mostly with SQL Server, which is cluster-aware, but applications you cluster don’t need to be. You can cluster any service, or resource on a server by just adding it to the cluster manager – the cluster manager will ensure it only runs on one server at a time, not allowing it to start on other nodes. For example, we use a monitoring tool that runs as a service – we installed the service on each cluster node and then added to the cluster manager – it now can be failed back and forth between nodes as a clustered resource, so it’s always online, is failure-resistant, and shares a segment of the HKLM in the registry between nodes – all without being explicitly cluster-aware.
SQL Server doesn’t need to be clustered when it’s installed on a cluster. While you can install a clustered instance of SQL Server (which automatically registers everything with the cluster manager), you can also install stand-alone instances of SQL Server (or any other application) on a cluster. That’s actually how a new feature in SQL 2012 – AlwaysOn – works: You install a non-clustered instance of SQL Server on different cluster nodes, and then you let the cluster manager coordinate client connections to the SQL Servers, but they still operate independantly and replicate their data between each other.

Hopefully this clears things up and doesn’t lead to more confusion. When I first started working with clustering, I had the impression that setting up a cluster caused the servers to act as one and share all their resources, but that understanding led to a lot of confusion when it came time to set something up or troubleshoot an issue. With the understanding that “clustered servers” are really just servers with clustered resources, and not actually clustered themselves, hopefully it will simply things!

Clustering walkthrough for SQL Server 2008 on Windows 2008

I recently stumbled across a great walk-through for clustering SQL Server on newer versions of Windows. It’s really thorough – everything from setting up iSCSI (in this case, to simulate a shared disk when it’s physically attached to one node – not ideal, but lets you test the walk-through), adding the required server roles, preparing the servers, and then a walkthrough of every screen in the SQL installation process. Thanks to the writer of this awesome blog!

http://dbperf.wordpress.com/2010/07/10/walkthrough-cluster-setup-sql-win-2008/

Slow SQL Server queries over a linked server from Windows Server 2008

We recently ran into an issue where linked server queries where sometimes taking significantly longer than they should have – queries that executed in under a second on the source server took two minutes when executed from a remote SQL Server over a linked server connection. We assumed it was something to do with our Windows Failover Cluster network connections, but it was happening on non-clustered servers as well.

After some digging, we discovered that it happened when the source server was Windows Server 2008 (or 2008 R2), regardless of what OS the remote linked server was running, and the issue wasn’t tied to any particular version of SQL Server. As it turns out, it related to everybody’s least favorite network setting, TCP Auto-Tuning – disabling it on the affected source servers resolved the problem immediately, and the queries returned instantly again.

To remedy the situation, follow these steps:

On the source SQL Server (the one executing the query, not the linked server), open an elevated command prompt (Right click on CMD.EXE and click “Run as Administrator”)
To disable TCP Auto-tuning, run the following command:
```
netsh int tcp set global autotuninglevel=disabled
```
To disable Windows Scaling Heuristics (another feature that can speed up clients, but is no good for servers), run:
```
netsh int tcp set heuristics disabled
```
This will automatically take effect after Windows has the chance to refresh some network communications – you can either wait about an hour, or, if you’d prefer it take effect immediately, restart the SQL Instance. A reboot isn’t necessary for this setting to take effect.

I’m not sure why these settings are on by default for servers – they seem to speed up internet browsing and other client-type activities (though there’s no shortage of complaints online about them grinding Windows Vista/7 clients to a halt as well), but can cause big problems on servers.

Accessing a clustered SQL Server instance without the instance name

When I clustered SQL Server 2005 the first time, it bothered me that I had to access each clustered instance using both the cluster DNS name and the instance name. If my SQL Cluster is called SQL-CLUSTER and the DNS alias of my first instance is SQL-INSTANCE1, I had to connect to SQL-INSTANCE1INSTANCE1. Since each instance has a dedicated name and IP address that uniquely identifies it, shouldn’t that be enough? Why is the instance name required?

It turns out it’s not! With a little tweaking, you can access the instance using just the SQL DNS name, without the instance name, and it works for as many SQL instances as you have on a cluster, not just one (you’ll need to repeat this process for each instance). Here’s how:

On the active node for the instance you’re adjusting, open the “SQL Server Configuration Manager” from the start menu.
Expand “SQL Server Network Configuration” and select the instance you want to edit.
In the right window, right-click “TCP/IP” and select “Properties.”
On the “Protocol” tab, ensure that “Listen All” is set to “Yes”
On the “IP Addresses” tab, scroll all the way to the bottom, to the “IPAll” section.
The “TCP port” box is blank by default – set it to 1433, the default port for SQL Server.
Click “OK” – you’ll receive a message that you need to restart the SQL Service.
Restart the SQL Service on that node.

Presto – done! You can now access this SQL Server instance using either “SQL-INSTANCE1\INSTANCE1” or just “SQL-INSTANCE”. It’s also worth noting that this network settings follows the instance between active nodes – if you failover the SQL instance to the other node, it should still be addressable the same way, and if you check the protocol settings, you’ll see that your change is active on that server now as well.

To me, it’s so much easier to not have to use the instance name when addressing the server, and since the service is responding on port 1433, the default port, this change should be compatible with every application that connects, and I haven’t had any problems at all.

If you try this and run into any problems, please leave a comment below.

UPDATE:

After playing with this a little more, it seems like it only works with the SQL Native Client. If you attempt to connect via the SQL Server OLEDB driver or the old SQL 2000 driver, you get the error that you’re not able to connect to the specified instance. Switching your connection method to the SQL Native Client (SQLNCLI) allows you to connect, assuming your application allows that. Has anybody else experienced this problem?