NexentaStor - increasing swap space

Here's another tip I forgot to post yesterday. How to increase the swap space on your NexentaStor server.

When I was trouble shooting our NexentaStore server when it has low available drive space, in the NexentaStor documentation I found that the default swap size is the memory size. For some reason we had a 1 gig swap file, and 4 gig of ram. Perhaps we didn't have all the ram populated on original install, so just to make sure everything was setup correct to help my trouble shooting, I went looking for a way to increase the swap space.

Since there wasn't any option I could find in the web gui, I was certain it was going to be a CLI thang. After some googling, I came across this:

http://www.crypticide.com/dropsafe/article/2649

I ssh-ed into the NexentaStor server as the admin user, and ran:

zfs set volsize=4G syspool/swap

This changed the swap space to the same size as the physical ram in the server.

NexentaStor hardware recommendations + Nexenta Support tip

I'd like to shout out to Dave who contacted me yesterday. Good to see people excited about this product, and it reminded me to keep posting up stuff on Nexenta and NexentaStor since it is a new product, and there isn't a heap of information out there yet.

I find great tips and solutions to problems daily via blogs and forums, so best I stay ontop of my work related blogging and give back to the community!

So part of this post is to document the hardware that I know works, and or, is in production with NexentaStor.

If you are currently researching the OpenSolaris HCL, it is a good start for picking out your hardware. From here, I would stick to this:

• Intel mobo and nic chipsets
• LSI controller cards

Here is a list of what I have in production with NexentaStor:

Motherboards:

• SuperMicro X7SBL-LN1

SAS/SATA Controllers:

• LSI SAS #SAS3081E Controller - set in JBOD mode

Drive Chassis:

• Supermicro SC836E1-R800B 16Bay Server Chassis (2 chassis daisy-chained via backplanes)
  
  • Coupled to Seagate ST31000340NS 1TB SATA drives

Network Cards:

•  Intel PRO/1000 PT Quad Port Server Adapter

Finding the correct cabling for backplanes and controller cards can be a real PITA sometimes, and I highly recommend you talk to these guys to get exactly what you need to hook up all the chassis, backplanes, and controller cards - http://www.cs-electronics.com

Finally, a Nexenta Support tip. If you wish to contact Nexenta support, the quickest and most efficient way to get to them is via the built in support ticketing system from within the web interface on your NexentaStor box (can also send one via command line). What it actually does, it will send Nexenta Support a breakdown of your setup as an attachment, and auto generates a ticket in their system. Don't bother emailing them, it takes much much longer that way - do it via the system they nicely provide coded right in the product.

Nexenta + NDMP Backup with BackupExec 12.5 + low free space issues

NexentaStor has been lots of fun to work with, there are still a few hickups as mentioned earlier in my blog, but it is turning out to be a pretty cool solution for clients that are not wanting to throw down big bucks on boutique NAS/SANes during this recession. Oh, and do I hear de-dupe is around the corner for ZFS?! :)

I was experimenting around with the NDMP Backup options in NexentaStor. I enabled the service, and punched in the details of our Symantec BackupExec 12.5 server, with NDMP option enabled.

Not being really familiar with this option in BackupExec, I just set everything pretty much with the default, and unfortunately, when browsing for backup devices, Nexenta does not appear in the NDMP devices, so off I went and logged a support ticket with Nexenta.

Here is their response, not holding my breath, but it would be seriously kick ass if NDMP backups did work with BackupExec (not to mention would get great speed, and can point backups to "latest" folders within a special backup snapshot for example - nice point in time backups/archives to tape!)

Looks like we do not have many customers using BackupExec. In general, this is our policy to support 3rd party software only if there is enough evidence that this is either a standard, or used by a significant number of users (which would make it de-facto standard). BackupExec is neither. We may be able though to get to it after the next major release 3.0 planned for end of November, early Dec.

Now onto free space in NexentaStor. I do encourage you to keep a close eye on available space on your volumes. We were in a situation where a slight mistake in a rsync script, filled up our Nexenta box. We had on or less than 10% space left, and it ended up with some strange behaviour.

Everything from no access via web interface, the console on the server locking up when trying to remove snapshots via the web interface, and errors and dead drives appearing after scrub jobs were issued (which weren't really dead drives!). After we blew away a bunch of stuff, and snapshots via command line (setup snapshot destroy), things settled down again.

I really like the weekly reports that the system can send to you, surf through them just to make sure available space is in check, and all should be fine.

NexentaStor 2.0/2.1 + Windows 2008 SP2 FAIL

Last week, I went ahead and upgraded a client's Windows 2008 domain controllers from SP1 to SP2. Reason, I'm always staying ontop keeping their Linux based NAS boxes working nicely with Active Directory etc, and thought it was a quiet time to update stuff across the board.

As it turns out, once Windows 2008 was at SP2 level, the NexentaStor 2.0 box now doesn't work on CIFS shares. The only way to access them is to use the "guest" login (a login box appears when you browse to it from the network).

Viewing the CIFS server log - got this:

Aug 20 13:27:37 server1 smbd[352]: [ID 526780 daemon.notice] Failed to establish NETLOGON credential chain
Aug 20 13:27:37 server1 smbd[352]: [ID 871254 daemon.error] smbd: failed joining domain.lan (UNSUCCESSFUL)
Aug 20 13:27:37 server1 idmap[335]: [ID 694198 daemon.notice] Configuration unchanged

After sending an email to support, they pointed me to:

http://www.nexenta.com/nexentastor-relnotes - Scroll down to v1.1.9 - known problem

and http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6850508

I have been told this will be addressed in September... or we can roll back to SP1 on the domain controllers! WTF, I guess this is Sun that is just slow to move on a pretty big bug. :(

NexentaStor - n00by tips and tricks

I now have a full licensed version of NexentaStor up and running for a client.

During the testing phase, I had run the trial software to make sure it had all the features as advertised and functioned in a way that would be practical for contractors like us, that prefer a slick solution that minimizes the tweaking around as much as possible!

Trail software was successfully run on:

• VMWare Player/Fusion
• Sun VirtualBox
• ASUS P5W DH Deluxe mobo - with additional PCIe Intel E1000 nic (onboard nics not supported)
• SuperMicro X6DVA series mobo

Had no problems joining it to a Windows 2003 Active Directory (mixed mode - with a legacy Windows 2000 DC), setting ACLs, snapshots, browsing the .zfs folder, etc, etc.

When it came to running on production hardware, we were unsuccessful getting it to run on:

• HP Proliant 160 G5
• We were able to install the Solaris nic drivers, however it would hang post install
• see: http://forums.opensolaris.com/message.jspa?messageID=2351
• suspect a video chipset issue!

Next we tried:

• SuperMicro X7SBL-LN1
• This was put into a Supermicro 4ru Jbod chassis - 16 x 1tb drives
• LSI SAS HBA - LSISAS3081E-R - set in jbod mode
• If you need help with SAS cables - these guys are great help: http://www.cs-electronics.com

No problems at all booting into NexentaStor now!

During this time, the production network did have a domain controller upgrade. The old Windows 2000 and Windows 2003 domain controllers were demoted and fresh new Windows 2008 domain controllers took over AD tasks for the network.

This is when I had the next challenge - I was able to get Nexentastor to join, and sometime NOT join the domain. Everything I did, could not get a AD user that was logged into a Windows box to access a CIFs share on NexentaStor.

After hitting a Nexenta support, DNF Storage support, and reading every detail in the Nexenta User Guide, I figured there was a problem with our AD upgrade. Soooo, if you have a domain joining issue, I urge you to check every bit of your DNS server setup in AD.

Even though in the forward zones, _msdcs.domain.lan all the relevant entries were correctly populated, the _msdcs zone within the domain.lan zone was incorrectly populated with old DCs entries.

Once this was created, everything was good to go.

One more gotcha... When setting ACLs on the CIF shares, don't just click every box in haste! If I did, I was still denied access to the share.

Set as follows:

• execute
• read
• create
• write
• delete
• but NO inherit

the inherit permission seemed to lock out access...