Life of an Expert

I had my first gig as an expert witness recently. Although a different setting, this video I found today was made even more enjoyable by the experience.

A Million Monkeys

Most have heard of the Infinite Monkey Theorem. I used to restate it during code review meetings as “a million monkeys could eventually code anything, but this code looks like it took three monkeys and three days.” The truth is, a lot of products have been poorly coded and although I am a strong believer in coding practices which optimize quality, there is no replacement for thorough verification.

An infinite number of monkeys may eventually write the complete works of Shakespeare but until it has been thoroughly evaluated as such, no one will ever know. The name  J. K. Rowling is familiar with many but does anyone know the team of editors and publishers who corrected the text, assisted in making it more readable and even identified that it was something of value in the first place?

To all you code writers, get out there and thank your verification teams.

Trick or Treat

It is not yet Halloween but its coming. This week I resurrected an old trick for a client and thought I would share it here as a technical/tutorial contribution instead of my usual rhetoric.

A great deal of engineering is capturing a process so that it can be modified and repeated to achieve a desired result. In electronics development, that generally means a series of invocations of various EDA programs of different sorts and from different vendors. Each of these invocations work on what is referred to as “source code” and create some intermediate representation of that code which is then fed into the next EDA invocation in the process and so on.

Along with the source code is generally a tool specific EDA setup file that specifies how the process should be completed. Typical source code might be Verilog, VHDL, C++, System C, System Verilog, etc. The method for capturing the process might be in a scripting language such as tcsh, bash, perl, python, tcl, etc. All of these languages have multiple methods for procedurally configuring the source code or the scripted process to achieve variations on the result from the same source code.

However, in many cases, the setup files for the actual EDA tools do not support variations in their invocation within the same setup file.

In fact, different setup files is often the expected method for achieving variation. So, the scripting language is often called up to choose from a selection of setup files based on the variation that only the script has the ability to understand. By selecting a setup file consistent with the desired configuration, a modification of how the EDA tool interprets the source code is achieved and its contribution to the variation in results is realized.

This often leads to a number of similar and somewhat redundant setup files that need to be maintained. Generally, the difference between two setup files is less than 50% of the content of the file. It could even be just one or two out of hundreds or thousands of lines of setup information.

This leads to a process maintenance nightmare where one desired change in setup that is common to all of the various setup files requires that they all be modified and verified to validate the process.

The trick to simplify the redundant maintenance of setup files is also simple, use one of the methods built into the source code files listed above. This method is the “text preprocessor.”

Specifically, I use the ‘C’ code preprocessor invoked by the command:
cpp -E -P -D<variationMacro1> -D<variationMacro2> … <setupFile> -o <temporarySetupFile>

This allows the use of all of the ‘C’ preprocessor constructs to be used in the setup file. The scripting language can invoke the ‘C’ preprocessor as shown above and then pass the temporarySetupFile to the EDA tool.

The only issue is when ‘C’ preprocessor constructs conflict with the syntax of the tool setup file. There are several ways to get around this but I find that using the ‘sed’ editor before and/or after “cpp” allows for management of ‘cpp’ sensitive characters like ‘#’, ‘/*’, and ‘//’.

For example replacing ‘#’ with ‘##’ in the setup file before ‘cpp’ and then reversing the replacement after preserves the ‘#’ character that might be needed in the setup file. Replacing ‘//’ with ‘%’ or some other special character keeps ‘cpp’ from interpreting what follows ‘//’ in the setup file as a comment. These edits will not work if the original file has ‘##’ or ‘%’ characters that should be preserved but I have found that by understanding the syntax of the setup file and the syntax of ‘cpp’ it is generally not difficult to work around any conflicts by using some innocuous character substitution. And if  ‘cpp’ does not meet your needs have a look at ‘m4’. I have never had to go there but if you do and you can then maybe you can write the next tutorial.

This trick can be used in more processes than just EDA development. It can be used in any textual based setup, source code, template, etc. to create customized variations based on  ‘cpp’ constructs.

So, there is the treat.

Eating an Elephant

Everyone knows how to eat an elephant.

Turns out you can eat a whale, a cow, or a chicken the same way and you don’t even have to know which animal it is.

Consulting is an interesting business, at least mine has been.

Sometimes you get to walk up to a clean whiteboard, capture the project goals and help a company design a whole new product. Maybe that’s rare for others, but I enjoyed that type of work all of my employed career and even a couple of times since beginning consulting.

At times, consulting work presents a problem that is in the middle or end of the development cycle and as a consultant you don’t necessarily get the opportunity to influence, or even fully appreciate the big picture. Often, to be efficient, a consultant must identify only as much of the context of the problem as to allow for the solution that the client has requested. In these scenarios the successful consultant needs to be able to quickly understand the environment and design practices of the client, identify the discrepancies, and propose or implement an improvement. More often than not, the challenge of the task is not the solution, but correctly identifying the accurate and optimum scope of the problem.

Lately, I have been providing assistance in ASIC emulation, also called rapid prototyping. In these cases, the actual product is generally near the end of its development cycle. It is generally a large, complex system with multiple processor cores, interfaces, memory systems, etc. There is generally firmware and hardware involved and an elaborate hardware/firmware/software design flow and integration. The focus on this type of assignment is rapidity. It would be impossible to be cost effective and try to understand the entire scope of the project.  So the consultant must quickly learn the environment, the design flow and only enough of the product details sufficient to assist in the process improvement as assigned.

His best tool is experience, which allows him the ability to quickly recognize and adapt to variations in process. For example, most ASIC design teams use version control. Amd. most currently use SVN, yet they all use SVN differently to achieve similar goals. An experienced consultant knows the goals, recognizes SVN, or its variant, or other RC methods and quickly understands and adapts to the methods and variations used by the client. The same is true for design flow scripting, their suite of EDA tools and even the organization of the clients design resources and systems network.

Consultants don’t always get to eat the whole animal and may not even know what kind of animal it is. He just does his part in taking bites and a good one takes bigger bites.

One Post A Month

It has been a month since my last posting at which time I changed the appearance of my site from one of the free wordpress “themes” to another. The old free one prominently showed the dates of my post but did not have the sidebar with author, follow widget and index information. My daughter designed my new business cards and I used the design as my new banner which did not go well with the old theme. I like the results of the new theme and new banner except information for when the post was released is a little more subtle. No matter, they are all as relevant today as they were when I posted them the first time.

I have a new contract starting next week. A fortune 500 company was given a referral to me by a previous client and once again my continued employment is owed to a colleague and not a fruit of my marketing skills. However client referral does speak to my reputation and marketable skills. The contract intends to be a rather short one so I will likely be ringing everyone’s bells again by the end of the year.

My “pro bono” work has still not been delivered. I have found that working on something even this simple when done intermittently and without a sense of urgency can lead to an even lesser quality product than I already would have suspected. I am doing better this week with an artificial deadline and persistent attention to the effort. I completely gutted and re-wrote half the code I had written and I am now back to testing. It is going much better and I intend to release it by the end of this week.

My “New Cool Project” has all but been abandoned for two reasons. First it received third priority after seeking compensated employment and the “pro bono” project and second it was based on “open source”. I hope to find time to post more on this later but I have a new de-appreciation for the “open source” fad. i have not completely given up on it but it has a very slim chance of getting any attention soon.

Rules of Debug and Testing

This week I am debugging code. I am doing some pro bono for a customer that asked for a test feature to be added to a product so that performance can be evaluated and improved. Rule #1, you can’t improve what you can’t measure. The customer has been good to me in the past and I am expecting that the result of the evaluation will possibly lead to new business. So, I am writing a SPI interface that will allow the product to offload a ridiculous amount of raw signal data that can be fed into an ideal system model, evaluated and compared to the product’s performance. Rule #2, compare multiple interpretations of a system and validate each individually. There really isn’t a “golden” model there are just multiple interpretations and verification is a process of evaluating consensus. Is the implementation wrong, is it the test or is it the presumption of desired behavior.

I spent an hour and described, in a document, the interface and the data stream format which will be captured by SPI to USB adapters connected to a PC and stored as raw data files on an HDD. Then I wrote the code in a couple of hours and installed it into the code database for the product. Since I architected the code database and already had a SPI slave module in the project library all I had to code was the state machine that captured the data,  formatted it and FIFO’d it to the SPI module. The stream is real-time and capture rate can be either slower or faster than the serial rate so the state machine and the data format has to handle both underflow and overflow conditions and the data captured does not match the transfer size of the SPI so the stream also has to be “packed”. And, since experience with the high speed adapter in the past has shown that transfer errors can occur when pushing the rate limit, a CRC is added to the stream at “block” intervals to protect data integrity.

Now it is time to debug my code, and that in any effort is N times more work than the concept or the implementation phase. Rule #3 schedule 2 units of time in concept, 1 unit of time in implementation and a minimum of 3 units of time in testing. When I was in charge of large ASIC designs, it often meant 2 to 6 months in concept which included a lot of detailed documentation, then 1 to 3 months writing code, followed by 6 to 18 months of testing. What I find is that if you skimp on the concept phase the implementation phase is longer due to re-visiting. And if you skimp on the last phase, which, once they hear that coding and initial testing is complete, is generally what upper management and marketing will push you to do, the product will fail miserably in the field. During the first phase design for test was always part of the product documentation and during the second phase I also allocated resources to developing a test plan for the product.

On this particular project the concept wasn’t totally new so I did skimp on phase 1 by leveraging past experience and so far I have got away with it in phase 2. However, because phase 1 and 2 went so fast I have a feeling phase 3 is going to take longer. This is not because I shorted the first 2 phases but because this particular rule is not going to scale to the smallness of this effort. In other words, I am probably looking at a 1-2-6 ratio where 1 unit is half of a day. Fortunately the original product already has an extensible design for test architecture and testbench so adding the new feature to that part of the database was no big effort.

Rule #4 Testing is an exponential problem. Thorough test grows exponential with the number of  modes and state. For example in this simple little add-on I have defined 8 modes for collecting the data and there is the overflow and underflow conditions as well as several exceptions to verify. Exceptions are conditions that the state machine can’t control and that are outside the list of defined conditions and expected operation. These conditions are often the most difficult to test and verify. I generally break the verification process into 3 steps, first is to get a basic mode operational. I pick the simplest scenario and write a test to find the easiest implementation failures, which usually include syntax and interface issues as well as initialization, idle and return to idle type issues. This is where I am after about a day of testing.

The second phase is a thorough test of defined modes and scenarios. Ideally these tests can be automated in both execution and evaluation. In an ongoing system development these become known as the regression test which is run on a snapshot of the database at a repeating schedule to maintain verified functionality is stable as the implementation evolves with features and corrections. This step is usually fairly quick to develop but time consuming execute. If you have the resources this can be parallelized, even with the other two steps.

The third step I refer to as stress testing. In this phase the boundaries of operation are explored for both proper operation and proper handling of exception scenarios. This step usually involves at least two types of testing, directed run-up to a boundary and randomized testing. Where boundaries of operation are known and have defined limits then testing that runs-up to the limit, crosses the limit and returns to within limits are written specifically. However with many modes of operation and many limits defined and many externally controlled inputs it is often difficult to prioritize the testing of every possible exception condition. It may even be difficult to test every possible proper operational condition. This is where randomized testing can be applied.

Randomized testing used to be the Holy Grail and was the part of testing that would get thrifted when marketing and budgetary pressures pushed. Now, however, due to the exponential rule it is replacing directed mode and boundary testing. This is what has driven the System Verilog language and the formal verification methodologies. The proper term might be constraint based testing and the difficulty is measuring coverage and tracing failures to root cause, especially in fault tolerant systems.

Even if you do a thorough job of testing, the product will fail in the field. Rule #5 all products have failures lurking regardless of the amount of testing or experience in the field. The concern is the likelihood, frequency, severity and recovery of the failures. I offer the following wager to anyone. $1 Million USD to anyone that can prove a product has no failure modes as long as they will match the wager if I can prove it has one. The value of a product can best be measured by the thoroughness of its errata sheet. We even see this today by people comparison shopping by looking at online customer reviews (really customer complaints). If a product has well documented complaints we can evaluate them and determine if those shortcoming affect how we are going to use (value) the product. Rule #6 Testing and documenting errata is often more valuable than fixing errata.

So, am I doing all three steps of testing including randomized testing on my SPI feature? Probably not, what do you expect for free? I am currently testing the basic mode and when done I will test all 16  mode scenarios, I might add it to my regression suite for the whole product and I will look at the most obvious exception modes, primarily transfer truncations due to unexpected negation of SPI slave select lines. I will leave randomized testing to my client in the field and he and I both will benefit from a re-programmable FPGA instead of a multi-million dollar ASIC investment. Rule #7 Product integrity is limited by product value.

 

Are Worker Status Laws Affecting the Contractor Market

I am curious and would love to hear from others with any experience regarding Worker Status laws. First, I have recently come across a couple of posts regarding a “crack down” on worker status violations. Probably in response to increased utilization of contracted workers as a result of recent legislated burdens in maintaining employees. Worker status is a determination of whether a worker must be classified as an employee or an independent contractor. The determination dictates whether the employer or the contractor bears the burden of employment taxes and worker protection under the law. Determination has always been a little gray and each tax and legal entity involved has their own methods. Further any difference in determination between the worker, workee* and taxing or legal entity is open for interpretation with the taxing or legal entity getting the final call.

Second, I am seeing a proliferation of so called W2 contract positions. My understanding is that these positions are offered as opposed to 1099 positions. Basically W2 means an employee, either employed by the workee or with an agency contracted to the workee, and 1099 means the position is a traditional independent contractor position (see more in the next paragraph), not many of these exist in my field anymore. This observation is consistent with my first observation that 1099 positions are being scrutinized and few 1099 positions would pass scrutiny. Basically the W2 positions are what I would call “temps” and these do not pay well. Surprisingly I have found that many recruiters don’t really understand the difference between a W2 and a 1099 position as they apply to the law and tax implications, especially the difference in cost and value to the worker and workee. I just recently tried to explain why the rate for a 1099 worker needs to be significantly higher than that for an equivalent W2 and why the cost to the workee is the same. He said he understood but still claimed he was not experiencing any difference in what was being offered.

Third, I have also been asked if I offer a corp-to-corp arrangement. Which, I discovered some time ago, is the term for exactly what I do offer. Provident Systems is an S-Corp which employs and pays me with a W2 and bears all of the employer burden of me, the employee of Provident Systems. A customer of Provident Systems gets a W9 Taxpayer ID as it would from any other vendor as required by IRS guidelines. And, clients do not even have to provide a 1099 to Provident Systems since 1099’s are not required to be provided to corporations, including S-corporations. So, a 1099 position, as opposed to a corp-to-corp, would be one in which the worker’s W9 indicted the worker as a “sole proprietor” and thus claiming his income on a schedule C of a personal 1040. In contrast, a corp-to-corp arrangement is not much different than buying goods and services as a company would from any other vendor.  Again, a distinction I have had trouble describing to recruiters looking for providers of short-term skilled services for their clients.  Part of their problem is the recruiting company they work for is looking to be the agency providing the W2 and then re-selling the worker’s time to the client. Trying to convert these opportunities to a corp-to-corp makes their compensation calculation difficult.

Finally, it is my understanding that since Provident Systems is an S-corporation which files a W2 for its employee and reports its income on a corporate tax return, there is no gray area on worker status with its clients. Much like there would be no gray area on a worker provided to a workee via a third party agency, such as the recruiting agencies. It just happens that in this case the “agency” providing the worker is also owned by the worker.

So, I would appreciate any comments on two areas. First, from HR professionals: are you seeing a push to only look for so called W2 contracts, aka temporary workers, through an agency? And second, from HR or legal professionals: is my information above correct? Is an employee of an S-corp (aka the owner) completely or mostly exempt from worker determination tests with respect to the S-corp’s clients?

* workee is to read as employer of contract client depending on worker status. Please excuse me for this liberty in terminology.